Nov 03 19:54:08.469719 ip-10-0-139-53 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 19:54:08.469733 ip-10-0-139-53 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 19:54:08.469742 ip-10-0-139-53 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 19:54:08.470053 ip-10-0-139-53 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 19:54:20.081878 ip-10-0-139-53 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 19:54:20.081892 ip-10-0-139-53 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e3331871b391470d88673aa693befd9c -- Nov 03 19:56:15.065453 ip-10-0-139-53 systemd[1]: Starting Kubernetes Kubelet... Nov 03 19:56:15.509367 ip-10-0-139-53 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:56:15.509367 ip-10-0-139-53 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 19:56:15.509367 ip-10-0-139-53 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:56:15.509367 ip-10-0-139-53 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:56:15.514489 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514406 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:15.514489 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514487 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:15.514489 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514491 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:15.514583 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514495 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:15.514583 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514498 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:15.514583 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514501 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:15.514583 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514504 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:15.514583 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514506 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:15.514583 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514509 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:15.514583 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514516 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:15.514583 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514520 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:15.514583 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514523 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:15.514583 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514525 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:15.514583 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514528 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:15.514583 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514531 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:15.514583 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514533 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:15.514583 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514536 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:15.514583 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514539 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:15.514583 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514542 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:15.514583 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514544 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:15.514583 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514547 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:15.514583 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514549 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:15.514583 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514552 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:15.515047 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514555 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:15.515047 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514559 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:15.515047 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514564 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:15.515047 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514567 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:15.515047 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514570 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:15.515047 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514573 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:15.515047 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514576 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:15.515047 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514579 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:15.515047 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514583 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:15.515047 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514587 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:15.515047 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514591 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:15.515047 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514594 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:15.515047 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514597 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:15.515047 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514599 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:15.515047 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514602 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:15.515047 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514605 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:15.515047 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514608 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:15.515047 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514610 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:15.515047 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514612 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:15.515516 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514615 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:15.515516 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514618 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:15.515516 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514621 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:15.515516 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514623 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:15.515516 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514625 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:15.515516 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514628 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:15.515516 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514630 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:15.515516 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514633 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:15.515516 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514635 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:15.515516 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514638 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:15.515516 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514640 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:15.515516 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514643 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:15.515516 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514646 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:15.515516 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514650 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:15.515516 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514653 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:15.515516 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514655 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:15.515516 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514658 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:15.515516 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514661 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:15.515516 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514663 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:15.515516 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514666 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:15.515989 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514669 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:15.515989 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514671 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:15.515989 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514674 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:15.515989 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514677 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:15.515989 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514679 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:15.515989 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514682 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:15.515989 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514684 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:15.515989 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514687 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:15.515989 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514690 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:15.515989 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514692 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:15.515989 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514695 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:15.515989 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514697 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:15.515989 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514700 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:15.515989 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514702 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:15.515989 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514705 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:15.515989 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514708 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:15.515989 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514710 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:15.515989 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514713 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:15.515989 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514716 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:15.515989 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514719 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:15.516520 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514722 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:15.516520 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514724 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:15.516520 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514727 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:15.516520 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514729 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:15.516520 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514733 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:15.516520 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514735 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:15.516520 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514738 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:15.516520 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514741 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:15.516520 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514744 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:15.516520 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514746 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:15.516520 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514749 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:15.516520 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514751 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:15.516520 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514754 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:15.516520 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514757 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:15.516520 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514759 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:15.516520 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514762 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:15.516520 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514764 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:15.516520 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514767 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:15.516520 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514769 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:15.516975 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514772 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:15.516975 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514774 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:15.516975 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514777 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:15.516975 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514781 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:15.516975 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.514784 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:15.516975 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515569 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:15.516975 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515576 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:15.516975 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515580 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:15.516975 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515583 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:15.516975 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515585 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:15.516975 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515606 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:15.516975 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515615 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:15.516975 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515620 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:15.516975 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515624 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:15.516975 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515627 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:15.516975 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515630 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:15.516975 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515633 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:15.516975 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515636 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:15.516975 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515638 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:15.517450 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515641 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:15.517450 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515643 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:15.517450 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515646 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:15.517450 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515649 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:15.517450 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515651 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:15.517450 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515653 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:15.517450 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515656 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:15.517450 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515659 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:15.517450 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515661 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:15.517450 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515664 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:15.517450 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515666 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:15.517450 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515669 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:15.517450 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515671 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:15.517450 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515674 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:15.517450 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515676 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:15.517450 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515679 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:15.517450 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515681 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:15.517450 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515684 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:15.517450 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515686 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:15.517450 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515689 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:15.517929 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515693 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:15.517929 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515697 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:15.517929 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515700 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:15.517929 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515702 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:15.517929 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515707 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:15.517929 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515710 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:15.517929 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515713 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:15.517929 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515716 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:15.517929 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515718 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:15.517929 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515721 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:15.517929 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515724 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:15.517929 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515726 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:15.517929 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515729 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:15.517929 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515731 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:15.517929 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515734 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:15.517929 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515737 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:15.517929 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515739 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:15.517929 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515742 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:15.517929 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515745 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:15.518579 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515747 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:15.518579 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515750 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:15.518579 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515752 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:15.518579 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515755 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:15.518579 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515757 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:15.518579 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515760 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:15.518579 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515763 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:15.518579 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515766 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:15.518579 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515769 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:15.518579 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515771 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:15.518579 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515773 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:15.518579 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515776 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:15.518579 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515779 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:15.518579 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515782 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:15.518579 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515784 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:15.518579 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515787 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:15.518579 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515790 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:15.518579 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515796 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:15.518579 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515799 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:15.519381 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515802 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:15.519381 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515804 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:15.519381 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515807 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:15.519381 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515809 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:15.519381 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515812 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:15.519381 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515814 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:15.519381 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515817 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:15.519381 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515820 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:15.519381 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515822 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:15.519381 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515825 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:15.519381 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515827 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:15.519381 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515830 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:15.519381 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515832 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:15.519381 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515835 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:15.519381 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515838 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:15.519381 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515840 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:15.519381 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515852 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:15.519381 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515855 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:15.519381 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515858 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:15.519381 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515860 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:15.519869 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515863 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:15.519869 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515866 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:15.519869 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515868 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:15.519869 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515871 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:15.519869 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515874 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:15.519869 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515877 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:15.519869 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515879 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:15.519869 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515882 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:15.519869 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515884 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:15.519869 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515888 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:15.519869 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515893 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:15.519869 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515896 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:15.519869 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515898 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:15.519869 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.515901 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:15.519869 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516373 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:15.519869 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516378 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:15.519869 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516381 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:15.519869 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516384 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:15.519869 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516387 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:15.520342 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516390 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:15.520342 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516392 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:15.520342 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516395 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:15.520342 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516398 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:15.520342 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516400 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:15.520342 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516403 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:15.520342 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516406 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:15.520342 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516408 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:15.520342 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516411 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:15.520342 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516414 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:15.520342 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516416 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:15.520342 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516419 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:15.520342 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516421 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:15.520342 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516424 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:15.520342 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516427 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:15.520342 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516429 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:15.520342 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516432 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:15.520342 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516435 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:15.520342 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516437 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:15.520342 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516440 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:15.520826 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516442 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:15.520826 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516445 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:15.520826 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516448 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:15.520826 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516453 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:15.520826 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516456 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:15.520826 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516459 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:15.520826 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516462 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:15.520826 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516464 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:15.520826 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516467 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:15.520826 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516469 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:15.520826 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516472 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:15.520826 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516474 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:15.520826 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516477 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:15.520826 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516480 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:15.520826 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516482 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:15.520826 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516485 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:15.520826 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516487 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:15.520826 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516490 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:15.520826 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516492 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:15.520826 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516495 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:15.521327 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516497 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:15.521327 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516500 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:15.521327 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516502 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:15.521327 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516505 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:15.521327 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516507 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:15.521327 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516510 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:15.521327 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516512 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:15.521327 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516515 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:15.521327 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516518 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:15.521327 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516520 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:15.521327 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516523 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:15.521327 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516525 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:15.521327 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516528 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:15.521327 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516531 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:15.521327 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516533 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:15.521327 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516539 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:15.521327 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516542 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:15.521327 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516545 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:15.521327 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516547 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:15.521773 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516551 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:15.521773 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516554 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:15.521773 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516556 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:15.521773 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516559 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:15.521773 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516561 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:15.521773 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516564 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:15.521773 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516566 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:15.521773 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516569 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:15.521773 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516572 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:15.521773 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516574 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:15.521773 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516577 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:15.521773 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516579 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:15.521773 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516582 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:15.521773 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516585 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:15.521773 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516588 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:15.521773 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516591 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:15.521773 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516594 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:15.521773 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516596 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:15.521773 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516598 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:15.522229 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516601 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:15.522229 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516603 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:15.522229 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516606 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:15.522229 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516608 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:15.522229 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516613 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:15.522229 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516617 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:15.522229 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516620 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:15.522229 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516623 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:15.522229 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516626 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:15.522229 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516631 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:15.522229 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516634 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:15.522229 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516637 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:15.522229 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516639 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:15.522229 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516643 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:15.522229 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516646 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:15.522229 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516648 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:15.522229 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516651 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:15.522229 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516654 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:15.522229 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516657 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516659 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516662 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516664 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.516667 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517823 2581 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517834 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517841 2581 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517846 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517851 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517854 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517859 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517863 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517866 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517869 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517873 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517876 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517879 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517882 2581 flags.go:64] FLAG: --cgroup-root="" Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517885 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517888 2581 flags.go:64] FLAG: --client-ca-file="" Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517891 2581 flags.go:64] FLAG: --cloud-provider="external" Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517894 2581 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517900 2581 flags.go:64] FLAG: --cluster-domain="" Nov 03 19:56:15.522673 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517903 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517906 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517910 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517913 2581 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517917 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517920 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517923 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517927 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517930 2581 flags.go:64] FLAG: --contention-profiling="false" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517933 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517937 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517940 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517943 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517947 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517950 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517953 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517956 2581 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517959 2581 flags.go:64] FLAG: --enable-server="true" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517962 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517966 2581 flags.go:64] FLAG: --event-burst="100" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517969 2581 flags.go:64] FLAG: --event-qps="50" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517972 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517976 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517980 2581 flags.go:64] FLAG: --eviction-hard="" Nov 03 19:56:15.523250 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517984 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517987 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517990 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517993 2581 flags.go:64] FLAG: --eviction-soft="" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517996 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.517999 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518002 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518006 2581 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518009 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518011 2581 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518045 2581 flags.go:64] FLAG: --feature-gates="" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518092 2581 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518801 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518809 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518816 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518823 2581 flags.go:64] FLAG: --healthz-port="10248" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518829 2581 flags.go:64] FLAG: --help="false" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518836 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518841 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518847 2581 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518859 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518864 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518870 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 19:56:15.523867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518875 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518880 2581 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518884 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518889 2581 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518894 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518904 2581 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518909 2581 flags.go:64] FLAG: --kube-reserved="" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518951 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518957 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518963 2581 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518968 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518973 2581 flags.go:64] FLAG: --lock-file="" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518977 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518982 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.518992 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519002 2581 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519007 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519012 2581 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519016 2581 flags.go:64] FLAG: --logging-format="text" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519021 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519027 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519032 2581 flags.go:64] FLAG: --manifest-url="" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519043 2581 flags.go:64] FLAG: --manifest-url-header="" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519051 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519056 2581 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 19:56:15.524454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519062 2581 flags.go:64] FLAG: --max-pods="110" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519067 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519071 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519076 2581 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519081 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519086 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519095 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519100 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519132 2581 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519139 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519146 2581 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519151 2581 flags.go:64] FLAG: --pod-cidr="" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519156 2581 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519162 2581 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519172 2581 flags.go:64] FLAG: --pods-per-core="0" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519219 2581 flags.go:64] FLAG: --port="10250" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519242 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519363 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0a62655d7020a54f6" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519369 2581 flags.go:64] FLAG: --qos-reserved="" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519375 2581 flags.go:64] FLAG: --read-only-port="10255" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519379 2581 flags.go:64] FLAG: --register-node="true" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519384 2581 flags.go:64] FLAG: --register-with-taints="" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519401 2581 flags.go:64] FLAG: --registry-burst="10" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519406 2581 flags.go:64] FLAG: --registry-qps="5" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519410 2581 flags.go:64] FLAG: --reserved-cpus="" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519415 2581 flags.go:64] FLAG: --reserved-memory="" Nov 03 19:56:15.525048 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519421 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519426 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519431 2581 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519436 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519449 2581 flags.go:64] FLAG: --runonce="false" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519454 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519462 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519467 2581 flags.go:64] FLAG: --seccomp-default="false" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519472 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519477 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519482 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519486 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519491 2581 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519496 2581 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519501 2581 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519505 2581 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519510 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519515 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519520 2581 flags.go:64] FLAG: --system-cgroups="" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519527 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519538 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519543 2581 flags.go:64] FLAG: --tls-cert-file="" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519549 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519557 2581 flags.go:64] FLAG: --tls-min-version="" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519561 2581 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 19:56:15.525680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519566 2581 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519571 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519578 2581 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519584 2581 flags.go:64] FLAG: --v="2" Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519594 2581 flags.go:64] FLAG: --version="false" Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519602 2581 flags.go:64] FLAG: --vmodule="" Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519608 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.519614 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519750 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519755 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519758 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519761 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519764 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519766 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519769 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519771 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519775 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519779 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519782 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519785 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519788 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:15.526420 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519791 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:15.527328 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519794 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:15.527328 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519797 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:15.527328 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519799 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:15.527328 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519802 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:15.527328 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519805 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:15.527328 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519808 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:15.527328 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519812 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:15.527328 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519816 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:15.527328 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519819 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:15.527328 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519822 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:15.527328 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519825 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:15.527328 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519828 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:15.527328 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519831 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:15.527328 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519833 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:15.527328 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519836 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:15.527328 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519839 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:15.527328 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519849 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:15.527328 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519853 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:15.527328 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519856 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:15.527328 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519859 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:15.528265 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519861 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:15.528265 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519864 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:15.528265 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519867 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:15.528265 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519870 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:15.528265 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519872 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:15.528265 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519875 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:15.528265 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519879 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:15.528265 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519882 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:15.528265 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519885 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:15.528265 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519887 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:15.528265 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519890 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:15.528265 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519894 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:15.528265 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519897 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:15.528265 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519899 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:15.528265 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519902 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:15.528265 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519905 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:15.528265 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519908 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:15.528265 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519911 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:15.528265 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519913 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:15.528898 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519916 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:15.528898 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519918 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:15.528898 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519922 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:15.528898 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519927 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:15.528898 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519930 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:15.528898 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519933 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:15.528898 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519935 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:15.528898 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519938 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:15.528898 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519940 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:15.528898 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519944 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:15.528898 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519946 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:15.528898 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519949 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:15.528898 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519952 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:15.528898 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519955 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:15.528898 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519957 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:15.528898 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519960 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:15.528898 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519963 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:15.528898 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519965 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:15.528898 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519968 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:15.529424 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519971 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:15.529424 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519974 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:15.529424 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519976 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:15.529424 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519979 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:15.529424 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519984 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:15.529424 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519988 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:15.529424 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519991 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:15.529424 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519993 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:15.529424 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519996 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:15.529424 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.519999 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:15.529424 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520002 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:15.529424 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520005 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:15.529424 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520008 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:15.529424 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520011 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:15.529424 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520014 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:15.529424 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520016 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:15.529424 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520019 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:15.529424 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520022 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:15.529424 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520024 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:15.530252 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520027 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:15.530252 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520029 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:15.530252 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520032 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:15.530252 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520035 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:15.530252 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520037 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:15.530252 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520040 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:15.530252 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520043 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:15.530252 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520045 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:15.530252 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520048 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:15.530252 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520050 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:15.530252 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520053 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:15.530252 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520055 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:15.530252 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520058 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:15.530252 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520061 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:15.530252 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520063 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:15.530252 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.520092 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:56:15.530946 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520136 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:15.530946 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520140 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:15.530946 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520147 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:15.530946 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520150 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:15.530946 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520152 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:15.530946 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520155 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:15.530946 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520158 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:15.530946 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520160 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:15.530946 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520163 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:15.530946 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520166 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:15.530946 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520169 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:15.530946 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520172 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:15.530946 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520187 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:15.530946 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520191 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:15.530946 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520194 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:15.530946 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520196 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:15.530946 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520199 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:15.530946 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520202 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:15.530946 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520205 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:15.531657 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520207 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:15.531657 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520210 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:15.531657 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520212 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:15.531657 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520215 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:15.531657 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520218 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:15.531657 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520220 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:15.531657 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520223 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:15.531657 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520225 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:15.531657 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520228 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:15.531657 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520230 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:15.531657 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520233 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:15.531657 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520236 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:15.531657 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520238 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:15.531657 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520241 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:15.531657 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520243 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:15.531657 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520248 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:15.531657 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520251 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:15.531657 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520254 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:15.531657 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520257 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:15.531657 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520260 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:15.532167 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520262 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:15.532167 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520265 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:15.532167 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520268 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:15.532167 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520271 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:15.532167 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520274 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:15.532167 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520277 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:15.532167 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520279 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:15.532167 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520282 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:15.532167 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520284 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:15.532167 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520287 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:15.532167 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520290 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:15.532167 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520292 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:15.532167 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520295 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:15.532167 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520297 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:15.532167 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520301 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:15.532167 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520305 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:15.532167 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520309 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:15.532167 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520312 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:15.532167 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520314 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:15.532954 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520317 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:15.532954 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520320 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:15.532954 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520322 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:15.532954 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520325 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:15.532954 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520327 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:15.532954 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520330 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:15.532954 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520332 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:15.532954 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520335 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:15.532954 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520340 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:15.532954 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520343 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:15.532954 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520345 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:15.532954 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520348 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:15.532954 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520351 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:15.532954 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520354 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:15.532954 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520356 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:15.532954 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520359 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:15.532954 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520362 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:15.532954 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520364 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:15.532954 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520367 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:15.532954 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520370 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:15.533761 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520377 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:15.533761 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520380 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:15.533761 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520383 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:15.533761 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520385 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:15.533761 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520388 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:15.533761 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520391 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:15.533761 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520394 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:15.533761 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520396 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:15.533761 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520399 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:15.533761 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520402 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:15.533761 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520405 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:15.533761 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520409 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:15.533761 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520412 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:15.533761 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520415 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:15.533761 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520417 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:15.533761 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520420 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:15.533761 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520422 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:15.533761 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520425 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:15.533761 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520428 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:15.534230 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520430 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:15.534230 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520435 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:15.534230 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520438 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:15.534230 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520440 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:15.534230 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520443 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:15.534230 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520446 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:15.534230 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520448 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:15.534230 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520451 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:15.534230 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.520454 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:15.534230 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.528591 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 19:56:15.534230 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.528611 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 19:56:15.534230 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.529356 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 19:56:15.534230 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.529367 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 19:56:15.534230 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529451 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:15.534230 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529460 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:15.534230 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529465 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:15.534230 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529470 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:15.534230 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529475 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:15.534230 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529479 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:15.534829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529484 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:15.534829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529488 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:15.534829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529493 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:15.534829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529497 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:15.534829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529501 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:15.534829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529506 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:15.534829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529511 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:15.534829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529515 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:15.534829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529519 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:15.534829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529523 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:15.534829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529528 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:15.534829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529532 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:15.534829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529536 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:15.534829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529541 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:15.534829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529545 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:15.534829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529550 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:15.534829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529554 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:15.534829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529558 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:15.534829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529562 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:15.534829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529566 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:15.535332 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529570 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:15.535332 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529574 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:15.535332 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529580 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:15.535332 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529586 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:15.535332 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529591 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:15.535332 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529595 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:15.535332 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529600 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:15.535332 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529604 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:15.535332 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529611 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:15.535332 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529616 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:15.535332 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529621 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:15.535332 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529626 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:15.535332 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529631 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:15.535332 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529635 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:15.535332 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529639 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:15.535332 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529644 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:15.535332 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529648 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:15.535332 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529652 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:15.535332 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529656 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:15.535829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529661 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:15.535829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529665 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:15.535829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529670 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:15.535829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529674 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:15.535829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529679 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:15.535829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529683 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:15.535829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529687 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:15.535829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529691 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:15.535829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529698 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:15.535829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529704 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:15.535829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529709 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:15.535829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529713 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:15.535829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529718 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:15.535829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529722 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:15.535829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529726 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:15.535829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529731 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:15.535829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529736 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:15.535829 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529742 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:15.536286 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529746 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:15.536286 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529751 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:15.536286 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529755 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:15.536286 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529759 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:15.536286 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529763 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:15.536286 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529768 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:15.536286 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529772 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:15.536286 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529776 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:15.536286 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529780 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:15.536286 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529784 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:15.536286 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529788 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:15.536286 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529793 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:15.536286 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529797 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:15.536286 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529802 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:15.536286 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529806 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:15.536286 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529810 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:15.536286 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529815 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:15.536286 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529820 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:15.536286 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529824 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:15.536286 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529828 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:15.536843 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529832 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:15.536843 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529836 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:15.536843 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529841 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:15.536843 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529845 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:15.536843 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529849 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:15.536843 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529854 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:15.536843 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529858 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:15.536843 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529862 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:15.536843 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529867 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:15.536843 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529871 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:15.536843 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529875 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:15.536843 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529879 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:15.536843 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529885 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:15.536843 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529890 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:15.536843 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529894 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:15.536843 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529898 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:15.536843 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529903 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:15.536843 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529907 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:15.536843 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529912 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:15.536843 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529916 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:15.537341 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529921 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:15.537341 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529925 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:15.537341 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.529929 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:15.537341 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.529971 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:56:15.537341 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530215 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:15.537341 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530226 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:15.537341 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530233 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:15.537341 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530238 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:15.537341 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530243 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:15.537341 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530247 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:15.537341 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530251 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:15.537341 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530255 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:15.537341 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530259 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:15.537341 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530264 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:15.537341 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530268 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:15.537341 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530272 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:15.537913 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530276 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:15.537913 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530280 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:15.537913 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530285 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:15.537913 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530289 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:15.537913 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530294 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:15.537913 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530298 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:15.537913 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530302 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:15.537913 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530307 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:15.537913 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530312 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:15.537913 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530317 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:15.537913 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530321 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:15.537913 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530325 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:15.537913 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530329 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:15.537913 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530333 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:15.537913 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530338 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:15.537913 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530342 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:15.537913 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530346 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:15.537913 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530350 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:15.537913 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530354 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:15.537913 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530358 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:15.538824 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530363 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:15.538824 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530366 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:15.538824 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530371 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:15.538824 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530375 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:15.538824 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530380 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:15.538824 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530384 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:15.538824 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530389 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:15.538824 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530393 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:15.538824 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530397 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:15.538824 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530401 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:15.538824 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530405 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:15.538824 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530409 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:15.538824 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530414 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:15.538824 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530418 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:15.538824 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530423 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:15.538824 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530427 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:15.538824 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530432 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:15.538824 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530436 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:15.538824 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530440 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:15.538824 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530444 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:15.539366 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530451 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:15.539366 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530458 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:15.539366 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530465 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:15.539366 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530469 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:15.539366 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530474 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:15.539366 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530479 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:15.539366 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530484 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:15.539366 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530488 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:15.539366 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530492 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:15.539366 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530497 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:15.539366 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530501 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:15.539366 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530505 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:15.539366 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530509 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:15.539366 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530514 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:15.539366 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530518 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:15.539366 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530522 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:15.539366 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530526 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:15.539366 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530530 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:15.539366 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530534 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:15.539817 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530538 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:15.539817 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530542 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:15.539817 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530547 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:15.539817 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530551 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:15.539817 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530555 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:15.539817 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530559 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:15.539817 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530564 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:15.539817 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530568 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:15.539817 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530573 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:15.539817 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530577 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:15.539817 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530581 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:15.539817 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530585 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:15.539817 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530590 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:15.539817 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530595 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:15.539817 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530599 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:15.539817 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530603 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:15.539817 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530608 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:15.539817 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530612 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:15.539817 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530616 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:15.540300 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530621 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:15.540300 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530625 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:15.540300 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530629 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:15.540300 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530633 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:15.540300 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530637 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:15.540300 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530641 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:15.540300 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530646 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:15.540300 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530650 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:15.540300 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530654 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:15.540300 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530658 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:15.540300 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530662 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:15.540300 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530666 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:15.540300 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530670 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:15.540300 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530675 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:15.540300 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530679 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:15.540300 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530683 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:15.540730 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.530723 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:56:15.540730 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530774 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:15.540730 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530781 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:15.540730 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530786 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:15.540730 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530791 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:15.540730 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530795 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:15.540730 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530799 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:15.540730 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530803 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:15.540730 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530808 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:15.540730 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530812 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:15.540730 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530816 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:15.540730 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530820 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:15.540730 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530825 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:15.540730 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530830 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:15.540730 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530841 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:15.540730 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530846 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:15.541394 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530850 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:15.541394 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530854 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:15.541394 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530858 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:15.541394 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530862 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:15.541394 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530866 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:15.541394 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530870 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:15.541394 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530875 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:15.541394 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530879 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:15.541394 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530883 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:15.541394 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530887 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:15.541394 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530894 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:15.541394 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530899 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:15.541394 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530903 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:15.541394 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530906 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:15.541394 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530911 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:15.541394 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530915 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:15.541394 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530920 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:15.541394 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530924 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:15.541394 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530928 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:15.541394 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530932 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:15.541939 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530936 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:15.541939 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530940 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:15.541939 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530944 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:15.541939 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530948 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:15.541939 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530954 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:15.541939 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530958 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:15.541939 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530962 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:15.541939 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530966 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:15.541939 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530970 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:15.541939 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530974 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:15.541939 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530981 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:15.541939 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530992 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:15.541939 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.530997 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:15.541939 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531002 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:15.541939 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531008 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:15.541939 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531012 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:15.541939 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531017 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:15.541939 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531021 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:15.541939 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531026 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:15.542446 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531030 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:15.542446 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531034 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:15.542446 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531038 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:15.542446 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531044 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:15.542446 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531048 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:15.542446 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531052 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:15.542446 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531057 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:15.542446 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531061 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:15.542446 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531065 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:15.542446 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531070 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:15.542446 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531074 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:15.542446 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531079 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:15.542446 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531083 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:15.542446 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531087 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:15.542446 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531091 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:15.542446 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531095 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:15.542446 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531099 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:15.542446 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531106 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:15.542446 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531110 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:15.542446 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531115 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:15.542947 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531119 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:15.542947 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531123 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:15.542947 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531128 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:15.542947 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531132 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:15.542947 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531137 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:15.542947 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531142 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:15.542947 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531147 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:15.542947 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531152 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:15.542947 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531156 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:15.542947 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531160 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:15.542947 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531165 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:15.542947 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531169 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:15.542947 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531173 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:15.542947 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531193 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:15.542947 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531197 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:15.542947 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531203 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:15.542947 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531208 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:15.542947 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531213 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:15.542947 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531217 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:15.543415 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531221 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:15.543415 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531225 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:15.543415 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531230 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:15.543415 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531234 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:15.543415 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531238 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:15.543415 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531242 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:15.543415 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531247 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:15.543415 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531256 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:15.543415 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531261 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:15.543415 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531266 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:15.543415 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531271 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:15.543415 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531276 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:15.543415 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:15.531281 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:15.543415 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.531367 2581 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 19:56:15.543415 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.534419 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 19:56:15.543415 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.535526 2581 server.go:1014] "Starting client certificate rotation" Nov 03 19:56:15.543415 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.535630 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:56:15.543415 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.536450 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:56:15.543865 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.538049 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:56:15.543865 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.541536 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:56:15.553122 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.553105 2581 log.go:25] "Validated CRI v1 runtime API" Nov 03 19:56:15.557069 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.557053 2581 log.go:25] "Validated CRI v1 image API" Nov 03 19:56:15.557910 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.557886 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 19:56:15.562129 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.562093 2581 fs.go:135] Filesystem UUIDs: map[43ddb09a-1a9b-4f99-9cb1-075c06fcec75:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a5e5e004-9e73-4ed8-91ad-730ff35b9edf:/dev/nvme0n1p3] Nov 03 19:56:15.562129 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.562112 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 19:56:15.566287 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.566161 2581 manager.go:217] Machine: {Timestamp:2026-11-03 19:56:15.565688954 +0000 UTC m=+0.371744815 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107986 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec282771c9f1812d2fbdc58594c4299e SystemUUID:ec282771-c9f1-812d-2fbd-c58594c4299e BootID:e3331871-b391-470d-8867-3aa693befd9c Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e6:f9:02:b3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e6:f9:02:b3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:62:be:ee:9f:be Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 19:56:15.567036 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.567021 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 19:56:15.567119 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.567100 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 19:56:15.570589 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.570562 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 19:56:15.570724 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.570589 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-53.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 19:56:15.570779 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.570726 2581 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 19:56:15.570779 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.570734 2581 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 19:56:15.570779 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.570750 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:56:15.570779 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.570768 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:56:15.572049 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.572038 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 19:56:15.573957 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.573941 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 19:56:15.576666 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.576654 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 19:56:15.576780 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.576771 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 19:56:15.576808 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.576803 2581 kubelet.go:493] "Attempting to sync node with API server" Nov 03 19:56:15.576834 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.576813 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 19:56:15.576834 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.576823 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 19:56:15.576834 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.576833 2581 kubelet.go:405] "Adding apiserver pod source" Nov 03 19:56:15.576935 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.576841 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 19:56:15.577227 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.577208 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:56:15.579768 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.579748 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 19:56:15.587022 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.587006 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:56:15.587947 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.587933 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 19:56:15.588010 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.587974 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 19:56:15.588150 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.588138 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 19:56:15.588225 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.588158 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 19:56:15.588225 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.588167 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 19:56:15.588225 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.588191 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 19:56:15.588225 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.588204 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 19:56:15.588225 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.588213 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 19:56:15.588878 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.588868 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 19:56:15.588924 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.588882 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 19:56:15.588924 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.588893 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 19:56:15.588924 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.588903 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 19:56:15.589125 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.588924 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 19:56:15.589125 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.588940 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 19:56:15.589891 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.589879 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 19:56:15.589957 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.589895 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 19:56:15.593963 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.593946 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-53.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 19:56:15.595459 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.595443 2581 server.go:1263] "Started kubelet" Nov 03 19:56:15.595604 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.595550 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 19:56:15.595665 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.595609 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 19:56:15.595665 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.595627 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 19:56:15.596329 ip-10-0-139-53 systemd[1]: Started Kubernetes Kubelet. Nov 03 19:56:15.597106 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.597089 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 19:56:15.597761 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.597744 2581 server.go:317] "Adding debug handlers to kubelet server" Nov 03 19:56:15.600887 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.600800 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 19:56:15.605957 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.605933 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 19:56:15.606029 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.605998 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-53.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 19:56:15.606092 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.606078 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 19:56:15.606136 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.606087 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 19:56:15.606203 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.606168 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-53.us-east-2.compute.internal\" not found" Nov 03 19:56:15.606870 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.606852 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 19:56:15.606870 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.606858 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 19:56:15.607022 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.606878 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 19:56:15.607022 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.606969 2581 reconstruct.go:97] "Volume reconstruction finished" Nov 03 19:56:15.607022 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.606979 2581 reconciler.go:29] "Reconciler: start to sync state" Nov 03 19:56:15.607165 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.605893 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-53.us-east-2.compute.internal.18e4a12f2d460a01 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-53.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-53.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:56:15.595399681 +0000 UTC m=+0.401455542,LastTimestamp:2026-11-03 19:56:15.595399681 +0000 UTC m=+0.401455542,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-53.us-east-2.compute.internal,}" Nov 03 19:56:15.609549 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.609526 2581 factory.go:55] Registering systemd factory Nov 03 19:56:15.609630 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.609554 2581 factory.go:223] Registration of the systemd container factory successfully Nov 03 19:56:15.609868 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.609851 2581 factory.go:153] Registering CRI-O factory Nov 03 19:56:15.609868 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.609869 2581 factory.go:223] Registration of the crio container factory successfully Nov 03 19:56:15.610003 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.609917 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 19:56:15.610003 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.609940 2581 factory.go:103] Registering Raw factory Nov 03 19:56:15.610003 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.609955 2581 manager.go:1199] Started watching for new ooms in manager Nov 03 19:56:15.611020 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.611008 2581 manager.go:319] Starting recovery of all containers Nov 03 19:56:15.611699 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.611669 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-53.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 19:56:15.614363 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.614337 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 19:56:15.620906 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.620885 2581 manager.go:324] Recovery completed Nov 03 19:56:15.625060 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.625029 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cc82c" Nov 03 19:56:15.627868 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.627854 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:15.629537 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.629522 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:15.629609 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.629554 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:15.629609 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.629569 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:15.630155 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.630133 2581 cpu_manager.go:226] "Starting" policy="none" Nov 03 19:56:15.630155 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.630145 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 19:56:15.630155 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.630148 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:15.630333 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.630162 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 19:56:15.630803 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.630792 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:15.630843 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.630814 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:15.630843 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.630824 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:15.632054 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.631969 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-53.us-east-2.compute.internal.18e4a12f2f4ef098 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-53.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-53.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-53.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:56:15.629537432 +0000 UTC m=+0.435593303,LastTimestamp:2026-11-03 19:56:15.629537432 +0000 UTC m=+0.435593303,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-53.us-east-2.compute.internal,}" Nov 03 19:56:15.634633 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.634615 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cc82c" Nov 03 19:56:15.647428 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.647336 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-53.us-east-2.compute.internal.18e4a12f2f4f4dbd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-53.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-53.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-53.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:56:15.629561277 +0000 UTC m=+0.435617146,LastTimestamp:2026-11-03 19:56:15.629561277 +0000 UTC m=+0.435617146,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-53.us-east-2.compute.internal,}" Nov 03 19:56:15.665484 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.664312 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-53.us-east-2.compute.internal.18e4a12f2f4f838b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-53.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-53.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-53.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:56:15.629575051 +0000 UTC m=+0.435630918,LastTimestamp:2026-11-03 19:56:15.629575051 +0000 UTC m=+0.435630918,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-53.us-east-2.compute.internal,}" Nov 03 19:56:15.678282 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.678214 2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-53.us-east-2.compute.internal.18e4a12f2f4ef098\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-53.us-east-2.compute.internal.18e4a12f2f4ef098 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-53.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-53.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-53.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:56:15.629537432 +0000 UTC m=+0.435593303,LastTimestamp:2026-11-03 19:56:15.630803176 +0000 UTC m=+0.436859037,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-53.us-east-2.compute.internal,}" Nov 03 19:56:15.691052 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.690981 2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-53.us-east-2.compute.internal.18e4a12f2f4f4dbd\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-53.us-east-2.compute.internal.18e4a12f2f4f4dbd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-53.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-53.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-53.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:56:15.629561277 +0000 UTC m=+0.435617146,LastTimestamp:2026-11-03 19:56:15.630818947 +0000 UTC m=+0.436874809,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-53.us-east-2.compute.internal,}" Nov 03 19:56:15.706503 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.706476 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-53.us-east-2.compute.internal\" not found" Nov 03 19:56:15.707328 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.707273 2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-53.us-east-2.compute.internal.18e4a12f2f4f838b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-53.us-east-2.compute.internal.18e4a12f2f4f838b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-53.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-53.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-53.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:56:15.629575051 +0000 UTC m=+0.435630918,LastTimestamp:2026-11-03 19:56:15.630828232 +0000 UTC m=+0.436884092,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-53.us-east-2.compute.internal,}" Nov 03 19:56:15.780170 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.780114 2581 policy_none.go:50] "Start" Nov 03 19:56:15.780170 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.780138 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 19:56:15.780170 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.780154 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 19:56:15.782326 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.782311 2581 policy_none.go:44] "Start" Nov 03 19:56:15.806914 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.806885 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-53.us-east-2.compute.internal\" not found" Nov 03 19:56:15.815741 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.815717 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 19:56:15.817977 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.817963 2581 manager.go:341] "Starting Device Plugin manager" Nov 03 19:56:15.818062 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.817995 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 19:56:15.818062 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.818006 2581 server.go:87] "Starting device plugin registration server" Nov 03 19:56:15.818449 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.818279 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 19:56:15.818509 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.818461 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 19:56:15.818598 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.818579 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 19:56:15.818682 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.818665 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 19:56:15.818682 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.818679 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 19:56:15.819204 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.819165 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 19:56:15.819299 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.819229 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-53.us-east-2.compute.internal\" not found" Nov 03 19:56:15.835681 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.835659 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 19:56:15.836225 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.836206 2581 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-53.us-east-2.compute.internal\" not found" node="ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:15.836946 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.836929 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 19:56:15.836990 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.836957 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 19:56:15.836990 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.836978 2581 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 19:56:15.837047 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.837020 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 19:56:15.840769 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.840752 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:15.918563 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.918535 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:15.919546 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.919530 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:15.919619 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.919563 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:15.919619 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.919578 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:15.919619 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.919601 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:15.928985 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.928971 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:15.929030 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.928991 2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-53.us-east-2.compute.internal\": node \"ip-10-0-139-53.us-east-2.compute.internal\" not found" Nov 03 19:56:15.937834 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.937805 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal"] Nov 03 19:56:15.937916 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.937864 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:15.938628 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.938608 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:15.938713 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.938637 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:15.938713 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.938647 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:15.940849 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.940837 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:15.940994 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.940981 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:15.941028 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.941010 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:15.941554 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.941537 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:15.941609 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.941569 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:15.941609 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.941579 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:15.941669 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.941540 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:15.941669 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.941641 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:15.941669 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.941654 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:15.943840 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.943824 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:15.943939 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.943855 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:15.944521 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.944505 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:15.944619 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.944538 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:15.944619 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:15.944550 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:15.957975 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.957952 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-53.us-east-2.compute.internal\" not found" Nov 03 19:56:15.971873 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.971850 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-53.us-east-2.compute.internal\" not found" node="ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:15.974573 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:15.974557 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-53.us-east-2.compute.internal\" not found" node="ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:16.009458 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.009440 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cb28595e81de1584c6e009472415b9b8-config\") pod \"kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal\" (UID: \"cb28595e81de1584c6e009472415b9b8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:16.009512 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.009462 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13c920c8ddfef32d3fa81fbcf93279d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal\" (UID: \"13c920c8ddfef32d3fa81fbcf93279d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:16.009512 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.009486 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13c920c8ddfef32d3fa81fbcf93279d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal\" (UID: \"13c920c8ddfef32d3fa81fbcf93279d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:16.058421 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:16.058369 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-53.us-east-2.compute.internal\" not found" Nov 03 19:56:16.109941 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.109916 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13c920c8ddfef32d3fa81fbcf93279d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal\" (UID: \"13c920c8ddfef32d3fa81fbcf93279d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:16.110012 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.109949 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cb28595e81de1584c6e009472415b9b8-config\") pod \"kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal\" (UID: \"cb28595e81de1584c6e009472415b9b8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:16.110012 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.109966 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13c920c8ddfef32d3fa81fbcf93279d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal\" (UID: \"13c920c8ddfef32d3fa81fbcf93279d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:16.110084 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.110027 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13c920c8ddfef32d3fa81fbcf93279d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal\" (UID: \"13c920c8ddfef32d3fa81fbcf93279d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:16.110084 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.110037 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cb28595e81de1584c6e009472415b9b8-config\") pod \"kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal\" (UID: \"cb28595e81de1584c6e009472415b9b8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:16.110084 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.110028 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13c920c8ddfef32d3fa81fbcf93279d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal\" (UID: \"13c920c8ddfef32d3fa81fbcf93279d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:16.159029 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:16.158997 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-53.us-east-2.compute.internal\" not found" Nov 03 19:56:16.259526 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:16.259489 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-53.us-east-2.compute.internal\" not found" Nov 03 19:56:16.275677 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.275655 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:16.277806 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.277791 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:16.361058 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:16.361025 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-53.us-east-2.compute.internal\" not found" Nov 03 19:56:16.461782 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:16.461734 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-53.us-east-2.compute.internal\" not found" Nov 03 19:56:16.517366 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.517339 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:16.535444 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.535419 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 19:56:16.535585 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.535564 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 19:56:16.535645 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.535603 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 19:56:16.562072 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:16.562041 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-53.us-east-2.compute.internal\" not found" Nov 03 19:56:16.662517 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:16.662433 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-53.us-east-2.compute.internal\" not found" Nov 03 19:56:16.763469 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:16.763438 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-53.us-east-2.compute.internal\" not found" Nov 03 19:56:16.765649 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.765627 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:16.766726 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.766708 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:16.766809 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.766747 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:16.766809 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.766763 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:16.783741 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.783652 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:51:15 +0000 UTC" deadline="2028-04-26 14:57:19.415538907 +0000 UTC" Nov 03 19:56:16.783741 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.783691 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12955h1m2.631851464s" Nov 03 19:56:16.863767 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:16.863738 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-53.us-east-2.compute.internal\" not found" Nov 03 19:56:16.940054 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:16.940010 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c920c8ddfef32d3fa81fbcf93279d1.slice/crio-676071769d6a0e543d6500b74fd35bf48763179597301db835c32ceb21c72e95 WatchSource:0}: Error finding container 676071769d6a0e543d6500b74fd35bf48763179597301db835c32ceb21c72e95: Status 404 returned error can't find the container with id 676071769d6a0e543d6500b74fd35bf48763179597301db835c32ceb21c72e95 Nov 03 19:56:16.946884 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.946864 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:56:16.949660 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:16.949635 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb28595e81de1584c6e009472415b9b8.slice/crio-5440d31e29ee75e3c31b392353d75842f76a00ab037294e19e4faa80f7f22d33 WatchSource:0}: Error finding container 5440d31e29ee75e3c31b392353d75842f76a00ab037294e19e4faa80f7f22d33: Status 404 returned error can't find the container with id 5440d31e29ee75e3c31b392353d75842f76a00ab037294e19e4faa80f7f22d33 Nov 03 19:56:16.964279 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:16.964256 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-53.us-east-2.compute.internal\" not found" Nov 03 19:56:16.965787 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:16.965773 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:17.007090 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.007070 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:17.025279 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.025258 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:56:17.026812 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.026799 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:17.035274 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.035258 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:56:17.049208 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.049182 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:17.577729 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.577701 2581 apiserver.go:52] "Watching apiserver" Nov 03 19:56:17.587890 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.587857 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 19:56:17.588655 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.588256 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal","openshift-dns/node-resolver-ksm7q","openshift-image-registry/node-ca-jjkqt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk","openshift-cluster-node-tuning-operator/tuned-5vpvg","openshift-multus/network-metrics-daemon-dcqpv","openshift-network-operator/iptables-alerter-fjr5q","openshift-ovn-kubernetes/ovnkube-node-bvlzl","kube-system/global-pull-secret-syncer-mvsdz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal","openshift-multus/multus-9gbmz","openshift-multus/multus-additional-cni-plugins-wjgg6","openshift-network-diagnostics/network-check-target-zn2ms","kube-system/konnectivity-agent-ntz8w"] Nov 03 19:56:17.591274 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.591244 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fjr5q" Nov 03 19:56:17.593482 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.593432 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:17.593658 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:17.593627 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2ms" podUID="f90a0a4e-2794-46fc-a799-9c1593d859ab" Nov 03 19:56:17.595819 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.595794 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jjkqt" Nov 03 19:56:17.596699 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.596659 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:56:17.596807 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.596755 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 19:56:17.596867 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.596857 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6tnlf\"" Nov 03 19:56:17.596921 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.596896 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 19:56:17.598528 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.598505 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.600693 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.600662 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.603371 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.603350 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:17.603468 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:17.603435 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dcqpv" podUID="ab34f1da-ace1-443d-bddd-9d9905d22e52" Nov 03 19:56:17.606015 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.605993 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:17.606114 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:17.606079 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvsdz" podUID="d2e4493e-d29f-49b6-851e-cbd01214bf5c" Nov 03 19:56:17.607200 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.607167 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 19:56:17.609620 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.609600 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ntz8w" Nov 03 19:56:17.612647 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.612608 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ksm7q" Nov 03 19:56:17.615750 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.615368 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.618978 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.618546 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.618978 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.618556 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-sysctl-conf\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.618978 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.618592 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-sys-fs\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.618978 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.618619 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d2e4493e-d29f-49b6-851e-cbd01214bf5c-dbus\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:17.618978 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.618646 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96087edc-f246-4ba4-a9d6-734a997da1f9-konnectivity-ca\") pod \"konnectivity-agent-ntz8w\" (UID: \"96087edc-f246-4ba4-a9d6-734a997da1f9\") " pod="kube-system/konnectivity-agent-ntz8w" Nov 03 19:56:17.618978 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.618714 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-sysctl-d\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.618978 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.618754 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.618978 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.618793 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96087edc-f246-4ba4-a9d6-734a997da1f9-agent-certs\") pod \"konnectivity-agent-ntz8w\" (UID: \"96087edc-f246-4ba4-a9d6-734a997da1f9\") " pod="kube-system/konnectivity-agent-ntz8w" Nov 03 19:56:17.618978 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.618817 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-modprobe-d\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.618978 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.618842 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-tuned\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.618978 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.618869 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs\") pod \"network-metrics-daemon-dcqpv\" (UID: \"ab34f1da-ace1-443d-bddd-9d9905d22e52\") " pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:17.618978 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.618896 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brzfd\" (UniqueName: \"kubernetes.io/projected/ab34f1da-ace1-443d-bddd-9d9905d22e52-kube-api-access-brzfd\") pod \"network-metrics-daemon-dcqpv\" (UID: \"ab34f1da-ace1-443d-bddd-9d9905d22e52\") " pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:17.618978 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.618973 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26ac4e9f-3445-4994-aa1b-086b39b8c1cc-host\") pod \"node-ca-jjkqt\" (UID: \"26ac4e9f-3445-4994-aa1b-086b39b8c1cc\") " pod="openshift-image-registry/node-ca-jjkqt" Nov 03 19:56:17.619680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619019 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a24da6cd-d805-4f8a-ac99-89dbd45c86c3-host-slash\") pod \"iptables-alerter-fjr5q\" (UID: \"a24da6cd-d805-4f8a-ac99-89dbd45c86c3\") " pod="openshift-network-operator/iptables-alerter-fjr5q" Nov 03 19:56:17.619680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619062 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsb75\" (UniqueName: \"kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75\") pod \"network-check-target-zn2ms\" (UID: \"f90a0a4e-2794-46fc-a799-9c1593d859ab\") " pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:17.619680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619109 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-kubernetes\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.619680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619135 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-run\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.619680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619179 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-var-lib-kubelet\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.619680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619208 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bqcz\" (UniqueName: \"kubernetes.io/projected/26ac4e9f-3445-4994-aa1b-086b39b8c1cc-kube-api-access-7bqcz\") pod \"node-ca-jjkqt\" (UID: \"26ac4e9f-3445-4994-aa1b-086b39b8c1cc\") " pod="openshift-image-registry/node-ca-jjkqt" Nov 03 19:56:17.619680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619257 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-registration-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.619680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619313 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-etc-selinux\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.619680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619340 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-lib-modules\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.619680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619364 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-host\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.619680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619481 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-socket-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.619680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619514 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgswz\" (UniqueName: \"kubernetes.io/projected/0d9796ba-d26b-494e-a4b1-2ebab6496f49-kube-api-access-pgswz\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.619680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619532 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d2e4493e-d29f-49b6-851e-cbd01214bf5c-kubelet-config\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:17.619680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619549 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a24da6cd-d805-4f8a-ac99-89dbd45c86c3-iptables-alerter-script\") pod \"iptables-alerter-fjr5q\" (UID: \"a24da6cd-d805-4f8a-ac99-89dbd45c86c3\") " pod="openshift-network-operator/iptables-alerter-fjr5q" Nov 03 19:56:17.619680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619595 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-sysconfig\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.619680 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619617 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-sys\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.620285 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619633 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk6q8\" (UniqueName: \"kubernetes.io/projected/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-kube-api-access-sk6q8\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.620285 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619658 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:17.620285 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619673 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jhkj\" (UniqueName: \"kubernetes.io/projected/a24da6cd-d805-4f8a-ac99-89dbd45c86c3-kube-api-access-5jhkj\") pod \"iptables-alerter-fjr5q\" (UID: \"a24da6cd-d805-4f8a-ac99-89dbd45c86c3\") " pod="openshift-network-operator/iptables-alerter-fjr5q" Nov 03 19:56:17.620285 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619699 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-systemd\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.620285 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619713 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-tmp\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.620285 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619733 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/26ac4e9f-3445-4994-aa1b-086b39b8c1cc-serviceca\") pod \"node-ca-jjkqt\" (UID: \"26ac4e9f-3445-4994-aa1b-086b39b8c1cc\") " pod="openshift-image-registry/node-ca-jjkqt" Nov 03 19:56:17.620285 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.619748 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-device-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.621997 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.621979 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.625167 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.625148 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 19:56:17.646248 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.646224 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gll4x" Nov 03 19:56:17.658355 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.658332 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gll4x" Nov 03 19:56:17.704953 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.704923 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 19:56:17.705485 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.705461 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 19:56:17.705646 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.705631 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 19:56:17.705742 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.705727 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lp2l4\"" Nov 03 19:56:17.705869 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.705849 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 19:56:17.705971 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.705947 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 19:56:17.706895 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.706778 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wsxsf\"" Nov 03 19:56:17.706895 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.706837 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 19:56:17.706895 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.706850 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 19:56:17.706895 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.706850 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 19:56:17.706895 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.706865 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 19:56:17.707281 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.707143 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 19:56:17.707281 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.707257 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n8st4\"" Nov 03 19:56:17.707399 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.707262 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 19:56:17.707399 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.707328 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 19:56:17.707399 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.707335 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 19:56:17.707399 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.707344 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 19:56:17.707399 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.707378 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l9cjs\"" Nov 03 19:56:17.707399 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.707382 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gc46m\"" Nov 03 19:56:17.707630 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.707417 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 19:56:17.707630 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.707465 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p44lh\"" Nov 03 19:56:17.707790 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.707773 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 19:56:17.708417 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.708397 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 19:56:17.708417 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.708411 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:56:17.708612 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.708438 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4n975\"" Nov 03 19:56:17.708612 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.708476 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 19:56:17.708612 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.708500 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 19:56:17.708612 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.708505 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 19:56:17.708612 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.708537 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 19:56:17.708612 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.708546 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 19:56:17.708612 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.708503 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 19:56:17.709117 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.709099 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 19:56:17.709207 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.709156 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5vrk4\"" Nov 03 19:56:17.720940 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.720914 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-var-lib-openvswitch\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.721054 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.720956 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.721054 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.720988 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-modprobe-d\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.721054 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721016 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-tuned\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.721054 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721044 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs\") pod \"network-metrics-daemon-dcqpv\" (UID: \"ab34f1da-ace1-443d-bddd-9d9905d22e52\") " pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:17.721303 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721071 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-cni-dir\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.721303 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721097 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-hostroot\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.721303 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721122 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/99713fae-ae74-4da5-b435-c73edc1c4843-env-overrides\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.721303 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721150 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctj28\" (UniqueName: \"kubernetes.io/projected/99713fae-ae74-4da5-b435-c73edc1c4843-kube-api-access-ctj28\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.721303 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721177 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26ac4e9f-3445-4994-aa1b-086b39b8c1cc-host\") pod \"node-ca-jjkqt\" (UID: \"26ac4e9f-3445-4994-aa1b-086b39b8c1cc\") " pod="openshift-image-registry/node-ca-jjkqt" Nov 03 19:56:17.721303 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721201 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-run-netns\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.721303 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721246 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-var-lib-kubelet\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.721303 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721273 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/88c8909b-9acd-46d0-9f19-d74e2db51f77-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.721759 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721309 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7vhv\" (UniqueName: \"kubernetes.io/projected/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40-kube-api-access-m7vhv\") pod \"node-resolver-ksm7q\" (UID: \"81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40\") " pod="openshift-dns/node-resolver-ksm7q" Nov 03 19:56:17.721759 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721336 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a24da6cd-d805-4f8a-ac99-89dbd45c86c3-host-slash\") pod \"iptables-alerter-fjr5q\" (UID: \"a24da6cd-d805-4f8a-ac99-89dbd45c86c3\") " pod="openshift-network-operator/iptables-alerter-fjr5q" Nov 03 19:56:17.721759 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721360 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-run\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.721759 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721388 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-var-lib-kubelet\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.721759 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721421 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-run-netns\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.721759 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721450 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bqcz\" (UniqueName: \"kubernetes.io/projected/26ac4e9f-3445-4994-aa1b-086b39b8c1cc-kube-api-access-7bqcz\") pod \"node-ca-jjkqt\" (UID: \"26ac4e9f-3445-4994-aa1b-086b39b8c1cc\") " pod="openshift-image-registry/node-ca-jjkqt" Nov 03 19:56:17.721759 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721475 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-registration-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.721759 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721499 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-etc-selinux\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.721759 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721539 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-host\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.721759 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721569 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-run-k8s-cni-cncf-io\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.721759 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721594 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-run-systemd\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.721759 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721631 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgswz\" (UniqueName: \"kubernetes.io/projected/0d9796ba-d26b-494e-a4b1-2ebab6496f49-kube-api-access-pgswz\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.721759 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721659 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d2e4493e-d29f-49b6-851e-cbd01214bf5c-kubelet-config\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:17.721759 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721686 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a24da6cd-d805-4f8a-ac99-89dbd45c86c3-iptables-alerter-script\") pod \"iptables-alerter-fjr5q\" (UID: \"a24da6cd-d805-4f8a-ac99-89dbd45c86c3\") " pod="openshift-network-operator/iptables-alerter-fjr5q" Nov 03 19:56:17.721759 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721722 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-sysconfig\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.721759 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721747 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-sys\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.722591 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721773 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-var-lib-cni-bin\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.722591 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721801 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-var-lib-cni-multus\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.722591 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721838 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/99713fae-ae74-4da5-b435-c73edc1c4843-ovnkube-script-lib\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.722591 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721875 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jhkj\" (UniqueName: \"kubernetes.io/projected/a24da6cd-d805-4f8a-ac99-89dbd45c86c3-kube-api-access-5jhkj\") pod \"iptables-alerter-fjr5q\" (UID: \"a24da6cd-d805-4f8a-ac99-89dbd45c86c3\") " pod="openshift-network-operator/iptables-alerter-fjr5q" Nov 03 19:56:17.722591 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721902 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-systemd\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.722591 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721930 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-tmp\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.722591 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721962 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/88c8909b-9acd-46d0-9f19-d74e2db51f77-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.722591 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.721992 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g8dn\" (UniqueName: \"kubernetes.io/projected/88c8909b-9acd-46d0-9f19-d74e2db51f77-kube-api-access-7g8dn\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.722591 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722018 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-slash\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.722591 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722043 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40-hosts-file\") pod \"node-resolver-ksm7q\" (UID: \"81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40\") " pod="openshift-dns/node-resolver-ksm7q" Nov 03 19:56:17.722591 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722068 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/26ac4e9f-3445-4994-aa1b-086b39b8c1cc-serviceca\") pod \"node-ca-jjkqt\" (UID: \"26ac4e9f-3445-4994-aa1b-086b39b8c1cc\") " pod="openshift-image-registry/node-ca-jjkqt" Nov 03 19:56:17.722591 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722095 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-device-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.722591 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722119 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-run-ovn\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.722591 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722145 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-sys-fs\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.722591 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722174 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96087edc-f246-4ba4-a9d6-734a997da1f9-konnectivity-ca\") pod \"konnectivity-agent-ntz8w\" (UID: \"96087edc-f246-4ba4-a9d6-734a997da1f9\") " pod="kube-system/konnectivity-agent-ntz8w" Nov 03 19:56:17.722591 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722201 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-daemon-config\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.722591 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722227 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/99713fae-ae74-4da5-b435-c73edc1c4843-ovnkube-config\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.723462 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722282 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.723462 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722309 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96087edc-f246-4ba4-a9d6-734a997da1f9-agent-certs\") pod \"konnectivity-agent-ntz8w\" (UID: \"96087edc-f246-4ba4-a9d6-734a997da1f9\") " pod="kube-system/konnectivity-agent-ntz8w" Nov 03 19:56:17.723462 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722340 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brzfd\" (UniqueName: \"kubernetes.io/projected/ab34f1da-ace1-443d-bddd-9d9905d22e52-kube-api-access-brzfd\") pod \"network-metrics-daemon-dcqpv\" (UID: \"ab34f1da-ace1-443d-bddd-9d9905d22e52\") " pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:17.723462 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722367 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6ggm\" (UniqueName: \"kubernetes.io/projected/f7dde7dd-245b-4437-9464-0e8b739caa98-kube-api-access-z6ggm\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.723462 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722395 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88c8909b-9acd-46d0-9f19-d74e2db51f77-system-cni-dir\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.723462 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722423 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88c8909b-9acd-46d0-9f19-d74e2db51f77-cni-binary-copy\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.723462 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722459 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88c8909b-9acd-46d0-9f19-d74e2db51f77-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.723462 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722487 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-systemd-units\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.723462 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722519 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsb75\" (UniqueName: \"kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75\") pod \"network-check-target-zn2ms\" (UID: \"f90a0a4e-2794-46fc-a799-9c1593d859ab\") " pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:17.723462 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722547 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-kubernetes\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.723462 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722575 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88c8909b-9acd-46d0-9f19-d74e2db51f77-cnibin\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.723462 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722599 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-kubelet\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.723462 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722636 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-log-socket\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.723462 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722663 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/99713fae-ae74-4da5-b435-c73edc1c4843-ovn-node-metrics-cert\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.723462 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722699 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40-tmp-dir\") pod \"node-resolver-ksm7q\" (UID: \"81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40\") " pod="openshift-dns/node-resolver-ksm7q" Nov 03 19:56:17.723462 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722727 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-lib-modules\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.724211 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722762 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7dde7dd-245b-4437-9464-0e8b739caa98-cni-binary-copy\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.724211 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722789 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-run-multus-certs\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.724211 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722819 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-etc-kubernetes\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.724211 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722847 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-etc-openvswitch\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.724211 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722873 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-cni-bin\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.724211 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722902 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-socket-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.724211 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722937 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk6q8\" (UniqueName: \"kubernetes.io/projected/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-kube-api-access-sk6q8\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.724211 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722963 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-node-log\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.724211 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.722993 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:17.724211 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723019 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-system-cni-dir\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.724211 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723046 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-conf-dir\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.724211 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723073 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-run-openvswitch\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.724211 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723103 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-run-ovn-kubernetes\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.724211 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723133 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-cni-netd\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.724211 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723158 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-cnibin\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.724211 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723187 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-sysctl-conf\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.724211 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723214 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d2e4493e-d29f-49b6-851e-cbd01214bf5c-dbus\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:17.724912 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723242 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-sysctl-d\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.724912 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723292 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-socket-dir-parent\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.724912 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723462 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-var-lib-kubelet\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.724912 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723486 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-kubernetes\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.724912 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723496 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-socket-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.724912 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723507 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-registration-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.724912 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723544 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-modprobe-d\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.724912 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723568 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-etc-selinux\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.724912 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:17.723576 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:17.724912 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723622 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-host\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.724912 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:17.723643 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs podName:ab34f1da-ace1-443d-bddd-9d9905d22e52 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:18.223620588 +0000 UTC m=+3.029676462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs") pod "network-metrics-daemon-dcqpv" (UID: "ab34f1da-ace1-443d-bddd-9d9905d22e52") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:17.724912 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723647 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-sysctl-d\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.724912 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723671 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a24da6cd-d805-4f8a-ac99-89dbd45c86c3-host-slash\") pod \"iptables-alerter-fjr5q\" (UID: \"a24da6cd-d805-4f8a-ac99-89dbd45c86c3\") " pod="openshift-network-operator/iptables-alerter-fjr5q" Nov 03 19:56:17.724912 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723753 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 19:56:17.724912 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723855 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d2e4493e-d29f-49b6-851e-cbd01214bf5c-dbus\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:17.724912 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:17.723883 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:17.724912 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723943 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-systemd\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.724912 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723977 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-sys-fs\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.725778 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723995 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-device-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.725778 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.723999 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d2e4493e-d29f-49b6-851e-cbd01214bf5c-kubelet-config\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:17.725778 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.724003 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.725778 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:17.724061 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret podName:d2e4493e-d29f-49b6-851e-cbd01214bf5c nodeName:}" failed. No retries permitted until 2026-11-03 19:56:18.22404602 +0000 UTC m=+3.030101892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret") pod "global-pull-secret-syncer-mvsdz" (UID: "d2e4493e-d29f-49b6-851e-cbd01214bf5c") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:17.725778 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.724118 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/26ac4e9f-3445-4994-aa1b-086b39b8c1cc-serviceca\") pod \"node-ca-jjkqt\" (UID: \"26ac4e9f-3445-4994-aa1b-086b39b8c1cc\") " pod="openshift-image-registry/node-ca-jjkqt" Nov 03 19:56:17.725778 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.724153 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-lib-modules\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.725778 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.724195 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-sys\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.725778 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.724237 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-sysconfig\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.725778 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.724280 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26ac4e9f-3445-4994-aa1b-086b39b8c1cc-host\") pod \"node-ca-jjkqt\" (UID: \"26ac4e9f-3445-4994-aa1b-086b39b8c1cc\") " pod="openshift-image-registry/node-ca-jjkqt" Nov 03 19:56:17.725778 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.724306 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-sysctl-conf\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.725778 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.724339 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-run\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.725778 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.724610 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a24da6cd-d805-4f8a-ac99-89dbd45c86c3-iptables-alerter-script\") pod \"iptables-alerter-fjr5q\" (UID: \"a24da6cd-d805-4f8a-ac99-89dbd45c86c3\") " pod="openshift-network-operator/iptables-alerter-fjr5q" Nov 03 19:56:17.725778 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.724611 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96087edc-f246-4ba4-a9d6-734a997da1f9-konnectivity-ca\") pod \"konnectivity-agent-ntz8w\" (UID: \"96087edc-f246-4ba4-a9d6-734a997da1f9\") " pod="kube-system/konnectivity-agent-ntz8w" Nov 03 19:56:17.728125 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.728106 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96087edc-f246-4ba4-a9d6-734a997da1f9-agent-certs\") pod \"konnectivity-agent-ntz8w\" (UID: \"96087edc-f246-4ba4-a9d6-734a997da1f9\") " pod="kube-system/konnectivity-agent-ntz8w" Nov 03 19:56:17.728345 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.728328 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-tmp\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.729404 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.729381 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-tuned\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.741448 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:17.741153 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:56:17.741448 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:17.741177 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:56:17.741448 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:17.741207 2581 projected.go:196] Error preparing data for projected volume kube-api-access-tsb75 for pod openshift-network-diagnostics/network-check-target-zn2ms: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:17.741448 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:17.741282 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75 podName:f90a0a4e-2794-46fc-a799-9c1593d859ab nodeName:}" failed. No retries permitted until 2026-11-03 19:56:18.241262588 +0000 UTC m=+3.047318458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tsb75" (UniqueName: "kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75") pod "network-check-target-zn2ms" (UID: "f90a0a4e-2794-46fc-a799-9c1593d859ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:17.744733 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.744707 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jhkj\" (UniqueName: \"kubernetes.io/projected/a24da6cd-d805-4f8a-ac99-89dbd45c86c3-kube-api-access-5jhkj\") pod \"iptables-alerter-fjr5q\" (UID: \"a24da6cd-d805-4f8a-ac99-89dbd45c86c3\") " pod="openshift-network-operator/iptables-alerter-fjr5q" Nov 03 19:56:17.744733 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.744729 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bqcz\" (UniqueName: \"kubernetes.io/projected/26ac4e9f-3445-4994-aa1b-086b39b8c1cc-kube-api-access-7bqcz\") pod \"node-ca-jjkqt\" (UID: \"26ac4e9f-3445-4994-aa1b-086b39b8c1cc\") " pod="openshift-image-registry/node-ca-jjkqt" Nov 03 19:56:17.744901 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.744738 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgswz\" (UniqueName: \"kubernetes.io/projected/0d9796ba-d26b-494e-a4b1-2ebab6496f49-kube-api-access-pgswz\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.745246 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.745196 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brzfd\" (UniqueName: \"kubernetes.io/projected/ab34f1da-ace1-443d-bddd-9d9905d22e52-kube-api-access-brzfd\") pod \"network-metrics-daemon-dcqpv\" (UID: \"ab34f1da-ace1-443d-bddd-9d9905d22e52\") " pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:17.746344 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.746288 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk6q8\" (UniqueName: \"kubernetes.io/projected/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-kube-api-access-sk6q8\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.823712 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.823671 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-run-k8s-cni-cncf-io\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.823893 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.823724 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-run-systemd\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.823893 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.823762 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-var-lib-cni-bin\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.823893 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.823787 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-var-lib-cni-multus\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.823893 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.823812 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/99713fae-ae74-4da5-b435-c73edc1c4843-ovnkube-script-lib\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.823893 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.823826 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-run-k8s-cni-cncf-io\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.823893 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.823843 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/88c8909b-9acd-46d0-9f19-d74e2db51f77-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.824201 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.823895 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g8dn\" (UniqueName: \"kubernetes.io/projected/88c8909b-9acd-46d0-9f19-d74e2db51f77-kube-api-access-7g8dn\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.824201 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.823927 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-slash\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.824201 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.823953 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40-hosts-file\") pod \"node-resolver-ksm7q\" (UID: \"81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40\") " pod="openshift-dns/node-resolver-ksm7q" Nov 03 19:56:17.824201 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.823978 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-run-ovn\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.824201 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824005 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-daemon-config\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.824201 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824030 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/99713fae-ae74-4da5-b435-c73edc1c4843-ovnkube-config\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.824201 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824059 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6ggm\" (UniqueName: \"kubernetes.io/projected/f7dde7dd-245b-4437-9464-0e8b739caa98-kube-api-access-z6ggm\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.824201 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824084 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88c8909b-9acd-46d0-9f19-d74e2db51f77-system-cni-dir\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.824201 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824109 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88c8909b-9acd-46d0-9f19-d74e2db51f77-cni-binary-copy\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.824201 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824135 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88c8909b-9acd-46d0-9f19-d74e2db51f77-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.824201 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824161 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-systemd-units\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.824784 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824220 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88c8909b-9acd-46d0-9f19-d74e2db51f77-cnibin\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.824784 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824246 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-kubelet\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.824784 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824271 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-log-socket\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.824784 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824298 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/99713fae-ae74-4da5-b435-c73edc1c4843-ovn-node-metrics-cert\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.824784 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824323 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40-tmp-dir\") pod \"node-resolver-ksm7q\" (UID: \"81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40\") " pod="openshift-dns/node-resolver-ksm7q" Nov 03 19:56:17.824784 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824350 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7dde7dd-245b-4437-9464-0e8b739caa98-cni-binary-copy\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.824784 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824375 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-run-multus-certs\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.824784 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824396 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-etc-kubernetes\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.824784 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824422 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-etc-openvswitch\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.824784 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824447 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-cni-bin\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.824784 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824464 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-node-log\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.824784 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824488 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-system-cni-dir\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.824784 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824504 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-conf-dir\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.824784 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824520 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-run-openvswitch\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.824784 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824536 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-run-ovn-kubernetes\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.824784 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824551 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-cni-netd\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.824784 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824567 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-cnibin\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.824784 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824593 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-socket-dir-parent\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.825674 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824615 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-var-lib-openvswitch\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.825674 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824673 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.825674 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824706 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-cni-dir\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.825674 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824728 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-hostroot\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.825674 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824744 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/99713fae-ae74-4da5-b435-c73edc1c4843-env-overrides\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.825674 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824760 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctj28\" (UniqueName: \"kubernetes.io/projected/99713fae-ae74-4da5-b435-c73edc1c4843-kube-api-access-ctj28\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.825674 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824787 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-run-netns\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.825674 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824806 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-var-lib-kubelet\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.825674 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824832 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/88c8909b-9acd-46d0-9f19-d74e2db51f77-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.825674 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824888 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7vhv\" (UniqueName: \"kubernetes.io/projected/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40-kube-api-access-m7vhv\") pod \"node-resolver-ksm7q\" (UID: \"81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40\") " pod="openshift-dns/node-resolver-ksm7q" Nov 03 19:56:17.825674 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824906 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-etc-kubernetes\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.825674 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824922 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-run-netns\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.825674 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.824939 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/88c8909b-9acd-46d0-9f19-d74e2db51f77-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.825674 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.825004 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-run-ovn\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.825674 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.825028 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40-hosts-file\") pod \"node-resolver-ksm7q\" (UID: \"81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40\") " pod="openshift-dns/node-resolver-ksm7q" Nov 03 19:56:17.825674 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.825095 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-etc-openvswitch\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.825674 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.825157 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-cni-bin\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.826515 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.825197 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-node-log\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.826515 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.825193 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-systemd-units\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.826515 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.825295 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-system-cni-dir\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.826515 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.825371 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-conf-dir\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.826515 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.825411 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-run-openvswitch\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.826515 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.825432 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-daemon-config\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.826515 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.825456 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-run-ovn-kubernetes\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.826515 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.825559 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-cni-netd\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.826515 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.825612 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-cnibin\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.826515 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.825660 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-socket-dir-parent\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.826515 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.825702 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-var-lib-openvswitch\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.826515 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.825742 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.826515 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.825794 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-cni-dir\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.826515 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.825834 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-hostroot\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.826515 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.825870 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/99713fae-ae74-4da5-b435-c73edc1c4843-ovnkube-config\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.826515 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.826138 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88c8909b-9acd-46d0-9f19-d74e2db51f77-system-cni-dir\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.826515 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.826233 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/99713fae-ae74-4da5-b435-c73edc1c4843-env-overrides\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.826515 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.826514 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-run-netns\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.827396 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.826541 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88c8909b-9acd-46d0-9f19-d74e2db51f77-cni-binary-copy\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.827396 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.826561 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-var-lib-kubelet\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.827396 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.826679 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-slash\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.827396 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.826925 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-run-netns\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.827396 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.826968 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-run-systemd\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.827396 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.826990 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-var-lib-cni-multus\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.827396 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.827019 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-var-lib-cni-bin\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.827396 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.827031 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/88c8909b-9acd-46d0-9f19-d74e2db51f77-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.827396 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.827077 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88c8909b-9acd-46d0-9f19-d74e2db51f77-cnibin\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.827396 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.827123 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-kubelet\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.827396 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.827153 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88c8909b-9acd-46d0-9f19-d74e2db51f77-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.827396 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.827155 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-log-socket\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.827999 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.827404 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-run-multus-certs\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.829578 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.828082 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40-tmp-dir\") pod \"node-resolver-ksm7q\" (UID: \"81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40\") " pod="openshift-dns/node-resolver-ksm7q" Nov 03 19:56:17.829578 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.828183 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/99713fae-ae74-4da5-b435-c73edc1c4843-ovnkube-script-lib\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.829578 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.828428 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7dde7dd-245b-4437-9464-0e8b739caa98-cni-binary-copy\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.834505 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.834481 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:17.835541 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.835507 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/99713fae-ae74-4da5-b435-c73edc1c4843-ovn-node-metrics-cert\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.840432 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.840405 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g8dn\" (UniqueName: \"kubernetes.io/projected/88c8909b-9acd-46d0-9f19-d74e2db51f77-kube-api-access-7g8dn\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:17.842067 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.842044 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7vhv\" (UniqueName: \"kubernetes.io/projected/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40-kube-api-access-m7vhv\") pod \"node-resolver-ksm7q\" (UID: \"81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40\") " pod="openshift-dns/node-resolver-ksm7q" Nov 03 19:56:17.843036 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.842958 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal" event={"ID":"cb28595e81de1584c6e009472415b9b8","Type":"ContainerStarted","Data":"5440d31e29ee75e3c31b392353d75842f76a00ab037294e19e4faa80f7f22d33"} Nov 03 19:56:17.843036 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.843016 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctj28\" (UniqueName: \"kubernetes.io/projected/99713fae-ae74-4da5-b435-c73edc1c4843-kube-api-access-ctj28\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.843400 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.843304 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6ggm\" (UniqueName: \"kubernetes.io/projected/f7dde7dd-245b-4437-9464-0e8b739caa98-kube-api-access-z6ggm\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.844649 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.844620 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" event={"ID":"13c920c8ddfef32d3fa81fbcf93279d1","Type":"ContainerStarted","Data":"676071769d6a0e543d6500b74fd35bf48763179597301db835c32ceb21c72e95"} Nov 03 19:56:17.911381 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.911350 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fjr5q" Nov 03 19:56:17.920688 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.920661 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jjkqt" Nov 03 19:56:17.923946 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:17.923896 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24da6cd_d805_4f8a_ac99_89dbd45c86c3.slice/crio-3451bfcb76a5e9ccc307b0620ef67625c59780543a4204f331a91b78057599c7 WatchSource:0}: Error finding container 3451bfcb76a5e9ccc307b0620ef67625c59780543a4204f331a91b78057599c7: Status 404 returned error can't find the container with id 3451bfcb76a5e9ccc307b0620ef67625c59780543a4204f331a91b78057599c7 Nov 03 19:56:17.930759 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:17.930732 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ac4e9f_3445_4994_aa1b_086b39b8c1cc.slice/crio-08ece5f0f14b1d4f47b31345c61311442bebb290188fbfdbb16987fa5037f038 WatchSource:0}: Error finding container 08ece5f0f14b1d4f47b31345c61311442bebb290188fbfdbb16987fa5037f038: Status 404 returned error can't find the container with id 08ece5f0f14b1d4f47b31345c61311442bebb290188fbfdbb16987fa5037f038 Nov 03 19:56:17.936981 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.936958 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:17.939341 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.939318 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:17.948711 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.948691 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ntz8w" Nov 03 19:56:17.949491 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:17.949469 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d9796ba_d26b_494e_a4b1_2ebab6496f49.slice/crio-f6294d9eff9fe195712f394713efe6f81e7a0d46bcbb1b544a7c758481901a3c WatchSource:0}: Error finding container f6294d9eff9fe195712f394713efe6f81e7a0d46bcbb1b544a7c758481901a3c: Status 404 returned error can't find the container with id f6294d9eff9fe195712f394713efe6f81e7a0d46bcbb1b544a7c758481901a3c Nov 03 19:56:17.951910 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:17.951831 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf838cffc_2fc7_4a3f_880b_8a80298cb1b4.slice/crio-29953c99f38e2e7838b4c07d9220b48b9f5728bc5d89f9e912b8141baba9db97 WatchSource:0}: Error finding container 29953c99f38e2e7838b4c07d9220b48b9f5728bc5d89f9e912b8141baba9db97: Status 404 returned error can't find the container with id 29953c99f38e2e7838b4c07d9220b48b9f5728bc5d89f9e912b8141baba9db97 Nov 03 19:56:17.956877 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.956789 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ksm7q" Nov 03 19:56:17.960562 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:17.960525 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96087edc_f246_4ba4_a9d6_734a997da1f9.slice/crio-72b25779bfb194395d3b79bb0ba2f8fc2ffe4b0e3b3f2dda6d6a6c02308c0c8b WatchSource:0}: Error finding container 72b25779bfb194395d3b79bb0ba2f8fc2ffe4b0e3b3f2dda6d6a6c02308c0c8b: Status 404 returned error can't find the container with id 72b25779bfb194395d3b79bb0ba2f8fc2ffe4b0e3b3f2dda6d6a6c02308c0c8b Nov 03 19:56:17.967844 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:17.967806 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ec8a45_bcb9_4aff_b9d0_a6bcf9fb2f40.slice/crio-c0def15a10438ae80cc43adfc56c708f394bb9e93cfd290dd1c406f282a41777 WatchSource:0}: Error finding container c0def15a10438ae80cc43adfc56c708f394bb9e93cfd290dd1c406f282a41777: Status 404 returned error can't find the container with id c0def15a10438ae80cc43adfc56c708f394bb9e93cfd290dd1c406f282a41777 Nov 03 19:56:17.983492 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.983466 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9gbmz" Nov 03 19:56:17.991178 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:17.991153 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:17.995089 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:17.995067 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7dde7dd_245b_4437_9464_0e8b739caa98.slice/crio-00cd428a9ec79e829670ad6140c6754ad0c64d61c98eff02f9bbfed9a43eb089 WatchSource:0}: Error finding container 00cd428a9ec79e829670ad6140c6754ad0c64d61c98eff02f9bbfed9a43eb089: Status 404 returned error can't find the container with id 00cd428a9ec79e829670ad6140c6754ad0c64d61c98eff02f9bbfed9a43eb089 Nov 03 19:56:18.002089 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:18.001927 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:18.003504 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:18.003473 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99713fae_ae74_4da5_b435_c73edc1c4843.slice/crio-b5d1ca779ca687573ee12ef853bfbf661e111e416c73c9103ef33a849cad0439 WatchSource:0}: Error finding container b5d1ca779ca687573ee12ef853bfbf661e111e416c73c9103ef33a849cad0439: Status 404 returned error can't find the container with id b5d1ca779ca687573ee12ef853bfbf661e111e416c73c9103ef33a849cad0439 Nov 03 19:56:18.012732 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:18.012710 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c8909b_9acd_46d0_9f19_d74e2db51f77.slice/crio-059ace63cca414c0b2cfdff9df65ba6bf009d06d374a5a43d5fc33edeb68c7d9 WatchSource:0}: Error finding container 059ace63cca414c0b2cfdff9df65ba6bf009d06d374a5a43d5fc33edeb68c7d9: Status 404 returned error can't find the container with id 059ace63cca414c0b2cfdff9df65ba6bf009d06d374a5a43d5fc33edeb68c7d9 Nov 03 19:56:18.191940 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:18.191866 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:18.228423 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:18.228387 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:18.228614 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:18.228448 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs\") pod \"network-metrics-daemon-dcqpv\" (UID: \"ab34f1da-ace1-443d-bddd-9d9905d22e52\") " pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:18.228614 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:18.228547 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:18.228614 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:18.228560 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:18.228614 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:18.228615 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs podName:ab34f1da-ace1-443d-bddd-9d9905d22e52 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:19.228596413 +0000 UTC m=+4.034652279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs") pod "network-metrics-daemon-dcqpv" (UID: "ab34f1da-ace1-443d-bddd-9d9905d22e52") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:18.228805 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:18.228631 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret podName:d2e4493e-d29f-49b6-851e-cbd01214bf5c nodeName:}" failed. No retries permitted until 2026-11-03 19:56:19.228623035 +0000 UTC m=+4.034678885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret") pod "global-pull-secret-syncer-mvsdz" (UID: "d2e4493e-d29f-49b6-851e-cbd01214bf5c") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:18.328962 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:18.328915 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsb75\" (UniqueName: \"kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75\") pod \"network-check-target-zn2ms\" (UID: \"f90a0a4e-2794-46fc-a799-9c1593d859ab\") " pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:18.329121 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:18.329087 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:56:18.329121 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:18.329117 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:56:18.329254 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:18.329132 2581 projected.go:196] Error preparing data for projected volume kube-api-access-tsb75 for pod openshift-network-diagnostics/network-check-target-zn2ms: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:18.329254 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:18.329214 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75 podName:f90a0a4e-2794-46fc-a799-9c1593d859ab nodeName:}" failed. No retries permitted until 2026-11-03 19:56:19.329182203 +0000 UTC m=+4.135238056 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tsb75" (UniqueName: "kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75") pod "network-check-target-zn2ms" (UID: "f90a0a4e-2794-46fc-a799-9c1593d859ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:18.662298 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:18.662021 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:51:17 +0000 UTC" deadline="2028-06-29 08:41:54.762731258 +0000 UTC" Nov 03 19:56:18.662298 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:18.662087 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14484h45m36.100649437s" Nov 03 19:56:18.873963 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:18.873923 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9gbmz" event={"ID":"f7dde7dd-245b-4437-9464-0e8b739caa98","Type":"ContainerStarted","Data":"00cd428a9ec79e829670ad6140c6754ad0c64d61c98eff02f9bbfed9a43eb089"} Nov 03 19:56:18.878926 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:18.878751 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ksm7q" event={"ID":"81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40","Type":"ContainerStarted","Data":"c0def15a10438ae80cc43adfc56c708f394bb9e93cfd290dd1c406f282a41777"} Nov 03 19:56:18.882310 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:18.882279 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ntz8w" event={"ID":"96087edc-f246-4ba4-a9d6-734a997da1f9","Type":"ContainerStarted","Data":"72b25779bfb194395d3b79bb0ba2f8fc2ffe4b0e3b3f2dda6d6a6c02308c0c8b"} Nov 03 19:56:18.886182 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:18.886151 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" event={"ID":"f838cffc-2fc7-4a3f-880b-8a80298cb1b4","Type":"ContainerStarted","Data":"29953c99f38e2e7838b4c07d9220b48b9f5728bc5d89f9e912b8141baba9db97"} Nov 03 19:56:18.889771 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:18.889680 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" event={"ID":"0d9796ba-d26b-494e-a4b1-2ebab6496f49","Type":"ContainerStarted","Data":"f6294d9eff9fe195712f394713efe6f81e7a0d46bcbb1b544a7c758481901a3c"} Nov 03 19:56:18.892809 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:18.892744 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jjkqt" event={"ID":"26ac4e9f-3445-4994-aa1b-086b39b8c1cc","Type":"ContainerStarted","Data":"08ece5f0f14b1d4f47b31345c61311442bebb290188fbfdbb16987fa5037f038"} Nov 03 19:56:18.894410 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:18.894339 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fjr5q" event={"ID":"a24da6cd-d805-4f8a-ac99-89dbd45c86c3","Type":"ContainerStarted","Data":"3451bfcb76a5e9ccc307b0620ef67625c59780543a4204f331a91b78057599c7"} Nov 03 19:56:18.896724 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:18.896669 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjgg6" event={"ID":"88c8909b-9acd-46d0-9f19-d74e2db51f77","Type":"ContainerStarted","Data":"059ace63cca414c0b2cfdff9df65ba6bf009d06d374a5a43d5fc33edeb68c7d9"} Nov 03 19:56:18.898239 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:18.898141 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" event={"ID":"99713fae-ae74-4da5-b435-c73edc1c4843","Type":"ContainerStarted","Data":"b5d1ca779ca687573ee12ef853bfbf661e111e416c73c9103ef33a849cad0439"} Nov 03 19:56:19.240644 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:19.240495 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs\") pod \"network-metrics-daemon-dcqpv\" (UID: \"ab34f1da-ace1-443d-bddd-9d9905d22e52\") " pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:19.240809 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:19.240664 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:19.240809 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:19.240802 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:19.240891 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:19.240862 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret podName:d2e4493e-d29f-49b6-851e-cbd01214bf5c nodeName:}" failed. No retries permitted until 2026-11-03 19:56:21.240843429 +0000 UTC m=+6.046899279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret") pod "global-pull-secret-syncer-mvsdz" (UID: "d2e4493e-d29f-49b6-851e-cbd01214bf5c") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:19.241292 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:19.241272 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:19.241392 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:19.241327 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs podName:ab34f1da-ace1-443d-bddd-9d9905d22e52 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:21.24131055 +0000 UTC m=+6.047366415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs") pod "network-metrics-daemon-dcqpv" (UID: "ab34f1da-ace1-443d-bddd-9d9905d22e52") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:19.341689 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:19.341101 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsb75\" (UniqueName: \"kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75\") pod \"network-check-target-zn2ms\" (UID: \"f90a0a4e-2794-46fc-a799-9c1593d859ab\") " pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:19.341689 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:19.341228 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:56:19.341689 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:19.341246 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:56:19.341689 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:19.341256 2581 projected.go:196] Error preparing data for projected volume kube-api-access-tsb75 for pod openshift-network-diagnostics/network-check-target-zn2ms: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:19.341689 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:19.341305 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75 podName:f90a0a4e-2794-46fc-a799-9c1593d859ab nodeName:}" failed. No retries permitted until 2026-11-03 19:56:21.341286553 +0000 UTC m=+6.147342425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tsb75" (UniqueName: "kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75") pod "network-check-target-zn2ms" (UID: "f90a0a4e-2794-46fc-a799-9c1593d859ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:19.662676 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:19.662558 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:51:17 +0000 UTC" deadline="2028-08-12 18:57:24.212576916 +0000 UTC" Nov 03 19:56:19.662676 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:19.662622 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15551h1m4.54995971s" Nov 03 19:56:19.847983 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:19.847470 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:19.847983 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:19.847605 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:19.847983 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:19.847606 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2ms" podUID="f90a0a4e-2794-46fc-a799-9c1593d859ab" Nov 03 19:56:19.847983 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:19.847686 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvsdz" podUID="d2e4493e-d29f-49b6-851e-cbd01214bf5c" Nov 03 19:56:19.847983 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:19.847742 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:19.847983 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:19.847877 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dcqpv" podUID="ab34f1da-ace1-443d-bddd-9d9905d22e52" Nov 03 19:56:19.921469 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:19.921132 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal" event={"ID":"cb28595e81de1584c6e009472415b9b8","Type":"ContainerStarted","Data":"fab49129cb63d95837398d02a8a0e4411b864b0e1e4b6c630e326b2582d5b903"} Nov 03 19:56:20.932496 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:20.932462 2581 generic.go:356] "Generic (PLEG): container finished" podID="13c920c8ddfef32d3fa81fbcf93279d1" containerID="3c09f2ddd018c8267a595104d4a4d2ca84b58de70c2aa17ac9750bc1cb1e1c1c" exitCode=0 Nov 03 19:56:20.933147 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:20.933005 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" event={"ID":"13c920c8ddfef32d3fa81fbcf93279d1","Type":"ContainerDied","Data":"3c09f2ddd018c8267a595104d4a4d2ca84b58de70c2aa17ac9750bc1cb1e1c1c"} Nov 03 19:56:20.951828 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:20.951768 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal" podStartSLOduration=3.951748223 podStartE2EDuration="3.951748223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:19.940956879 +0000 UTC m=+4.747012752" watchObservedRunningTime="2026-11-03 19:56:20.951748223 +0000 UTC m=+5.757804096" Nov 03 19:56:21.255668 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:21.254973 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:21.255668 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:21.255027 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs\") pod \"network-metrics-daemon-dcqpv\" (UID: \"ab34f1da-ace1-443d-bddd-9d9905d22e52\") " pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:21.255668 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:21.255147 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:21.255668 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:21.255224 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs podName:ab34f1da-ace1-443d-bddd-9d9905d22e52 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:25.255204728 +0000 UTC m=+10.061260599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs") pod "network-metrics-daemon-dcqpv" (UID: "ab34f1da-ace1-443d-bddd-9d9905d22e52") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:21.255668 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:21.255294 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:21.255668 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:21.255328 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret podName:d2e4493e-d29f-49b6-851e-cbd01214bf5c nodeName:}" failed. No retries permitted until 2026-11-03 19:56:25.255317503 +0000 UTC m=+10.061373351 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret") pod "global-pull-secret-syncer-mvsdz" (UID: "d2e4493e-d29f-49b6-851e-cbd01214bf5c") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:21.355641 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:21.355562 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsb75\" (UniqueName: \"kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75\") pod \"network-check-target-zn2ms\" (UID: \"f90a0a4e-2794-46fc-a799-9c1593d859ab\") " pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:21.355883 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:21.355863 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:56:21.355941 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:21.355888 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:56:21.355941 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:21.355903 2581 projected.go:196] Error preparing data for projected volume kube-api-access-tsb75 for pod openshift-network-diagnostics/network-check-target-zn2ms: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:21.356015 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:21.355958 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75 podName:f90a0a4e-2794-46fc-a799-9c1593d859ab nodeName:}" failed. No retries permitted until 2026-11-03 19:56:25.355942578 +0000 UTC m=+10.161998650 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tsb75" (UniqueName: "kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75") pod "network-check-target-zn2ms" (UID: "f90a0a4e-2794-46fc-a799-9c1593d859ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:21.839638 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:21.839562 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:21.839850 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:21.839686 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2ms" podUID="f90a0a4e-2794-46fc-a799-9c1593d859ab" Nov 03 19:56:21.840083 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:21.840065 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:21.840238 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:21.840191 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dcqpv" podUID="ab34f1da-ace1-443d-bddd-9d9905d22e52" Nov 03 19:56:21.840238 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:21.840231 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:21.840353 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:21.840297 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvsdz" podUID="d2e4493e-d29f-49b6-851e-cbd01214bf5c" Nov 03 19:56:21.944620 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:21.943932 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" event={"ID":"13c920c8ddfef32d3fa81fbcf93279d1","Type":"ContainerStarted","Data":"1638aced06b221e3add91dba0bd53e486df3bd54113e51010e31606bce114efa"} Nov 03 19:56:21.948080 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:21.947235 2581 generic.go:356] "Generic (PLEG): container finished" podID="88c8909b-9acd-46d0-9f19-d74e2db51f77" containerID="c7d5d96db7a890be66ed19320e9d376743ea9bcc10db797e33b5cfc7940130a7" exitCode=0 Nov 03 19:56:21.948080 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:21.947270 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjgg6" event={"ID":"88c8909b-9acd-46d0-9f19-d74e2db51f77","Type":"ContainerDied","Data":"c7d5d96db7a890be66ed19320e9d376743ea9bcc10db797e33b5cfc7940130a7"} Nov 03 19:56:21.962843 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:21.962601 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" podStartSLOduration=4.962583078 podStartE2EDuration="4.962583078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:21.961692433 +0000 UTC m=+6.767748305" watchObservedRunningTime="2026-11-03 19:56:21.962583078 +0000 UTC m=+6.768638949" Nov 03 19:56:22.954918 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:22.954861 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ntz8w" event={"ID":"96087edc-f246-4ba4-a9d6-734a997da1f9","Type":"ContainerStarted","Data":"2dc51c52c6fdcd27acb84978fb5b6d0e41d9992142d9f7b3992906f67f28d49f"} Nov 03 19:56:22.972786 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:22.972294 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ntz8w" podStartSLOduration=2.362331286 podStartE2EDuration="6.97227558s" totalImagesPullingTime="4.609944294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:22.971698228 +0000 UTC m=+7.777754095" watchObservedRunningTime="2026-11-03 19:56:22.97227558 +0000 UTC m=+7.778331452" Nov 03 19:56:23.840245 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:23.840165 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:23.840404 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:23.840298 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvsdz" podUID="d2e4493e-d29f-49b6-851e-cbd01214bf5c" Nov 03 19:56:23.840633 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:23.840609 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:23.840725 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:23.840677 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2ms" podUID="f90a0a4e-2794-46fc-a799-9c1593d859ab" Nov 03 19:56:23.840790 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:23.840727 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:23.840790 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:23.840780 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dcqpv" podUID="ab34f1da-ace1-443d-bddd-9d9905d22e52" Nov 03 19:56:23.958249 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:23.958207 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" event={"ID":"0d9796ba-d26b-494e-a4b1-2ebab6496f49","Type":"ContainerStarted","Data":"d9df517050f9a26113244140f8f95b09ecc77dc5a112d4e021811c05bc972830"} Nov 03 19:56:23.960590 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:23.960569 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jjkqt" event={"ID":"26ac4e9f-3445-4994-aa1b-086b39b8c1cc","Type":"ContainerStarted","Data":"aae46a6c4ceb7bd388850a8ac6832aac2ba5bd715c3ae1225e99521d34dbac2c"} Nov 03 19:56:25.124744 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:25.124374 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 19:56:25.205338 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:25.205269 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ntz8w" Nov 03 19:56:25.299877 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:25.298322 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:25.299877 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:25.298376 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs\") pod \"network-metrics-daemon-dcqpv\" (UID: \"ab34f1da-ace1-443d-bddd-9d9905d22e52\") " pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:25.299877 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:25.298547 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:25.299877 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:25.298614 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs podName:ab34f1da-ace1-443d-bddd-9d9905d22e52 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:33.298594456 +0000 UTC m=+18.104650307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs") pod "network-metrics-daemon-dcqpv" (UID: "ab34f1da-ace1-443d-bddd-9d9905d22e52") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:25.299877 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:25.299743 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:25.299877 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:25.299836 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret podName:d2e4493e-d29f-49b6-851e-cbd01214bf5c nodeName:}" failed. No retries permitted until 2026-11-03 19:56:33.299819516 +0000 UTC m=+18.105875378 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret") pod "global-pull-secret-syncer-mvsdz" (UID: "d2e4493e-d29f-49b6-851e-cbd01214bf5c") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:25.399288 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:25.399250 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsb75\" (UniqueName: \"kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75\") pod \"network-check-target-zn2ms\" (UID: \"f90a0a4e-2794-46fc-a799-9c1593d859ab\") " pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:25.399445 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:25.399401 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:56:25.399445 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:25.399424 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:56:25.399445 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:25.399437 2581 projected.go:196] Error preparing data for projected volume kube-api-access-tsb75 for pod openshift-network-diagnostics/network-check-target-zn2ms: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:25.399607 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:25.399497 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75 podName:f90a0a4e-2794-46fc-a799-9c1593d859ab nodeName:}" failed. No retries permitted until 2026-11-03 19:56:33.39947835 +0000 UTC m=+18.205534221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tsb75" (UniqueName: "kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75") pod "network-check-target-zn2ms" (UID: "f90a0a4e-2794-46fc-a799-9c1593d859ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:25.829195 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:25.824424 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:56:25.124390209Z","UUID":"24805c0e-bf14-4772-a7c4-36136b491ae2","Handler":null,"Name":"","Endpoint":""} Nov 03 19:56:25.831058 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:25.830727 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 19:56:25.831058 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:25.830808 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 19:56:25.847347 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:25.847321 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:25.847622 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:25.847598 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvsdz" podUID="d2e4493e-d29f-49b6-851e-cbd01214bf5c" Nov 03 19:56:25.848079 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:25.848061 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:25.848293 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:25.848274 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2ms" podUID="f90a0a4e-2794-46fc-a799-9c1593d859ab" Nov 03 19:56:25.848600 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:25.848580 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:25.848844 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:25.848825 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dcqpv" podUID="ab34f1da-ace1-443d-bddd-9d9905d22e52" Nov 03 19:56:25.974968 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:25.974004 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9gbmz" event={"ID":"f7dde7dd-245b-4437-9464-0e8b739caa98","Type":"ContainerStarted","Data":"697d23a9def0869192ca4636fd8ccd3edd2be1689f3c88182b151c1d653cfb16"} Nov 03 19:56:25.980428 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:25.980403 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" event={"ID":"0d9796ba-d26b-494e-a4b1-2ebab6496f49","Type":"ContainerStarted","Data":"036cb279b7991031f7dd007cf0a380a03b7a01afa80f6e6b04aee342fb3d8425"} Nov 03 19:56:25.996510 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:25.996079 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jjkqt" podStartSLOduration=4.999513018 podStartE2EDuration="9.996067104s" totalImagesPullingTime="4.996554086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:23.979700464 +0000 UTC m=+8.785756335" watchObservedRunningTime="2026-11-03 19:56:25.996067104 +0000 UTC m=+10.802123026" Nov 03 19:56:26.352983 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:26.352948 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ntz8w" Nov 03 19:56:26.353784 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:26.353730 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ntz8w" Nov 03 19:56:26.371165 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:26.371127 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9gbmz" podStartSLOduration=2.722596527 podStartE2EDuration="10.371112985s" totalImagesPullingTime="7.648516458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:25.996496732 +0000 UTC m=+10.802552602" watchObservedRunningTime="2026-11-03 19:56:26.371112985 +0000 UTC m=+11.177168851" Nov 03 19:56:26.985318 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:26.985017 2581 generic.go:356] "Generic (PLEG): container finished" podID="88c8909b-9acd-46d0-9f19-d74e2db51f77" containerID="a1905c04f9a2c1037bdea9d29299cd8cb1ba04936eb8e822d67b5d1332c6f26e" exitCode=0 Nov 03 19:56:26.985465 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:26.985195 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjgg6" event={"ID":"88c8909b-9acd-46d0-9f19-d74e2db51f77","Type":"ContainerDied","Data":"a1905c04f9a2c1037bdea9d29299cd8cb1ba04936eb8e822d67b5d1332c6f26e"} Nov 03 19:56:26.987156 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:26.987135 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ksm7q" event={"ID":"81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40","Type":"ContainerStarted","Data":"6f37629d92e6c17f1360350512890b65d5670786349b750609284b3b2f823964"} Nov 03 19:56:26.994499 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:26.991344 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" event={"ID":"0d9796ba-d26b-494e-a4b1-2ebab6496f49","Type":"ContainerStarted","Data":"8d09446d74f4c928d2757815f03fbf06e4afcccdb4675f849ca172951e10b288"} Nov 03 19:56:26.994499 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:26.991775 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ntz8w" Nov 03 19:56:27.032937 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:27.032893 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" podStartSLOduration=2.955327815 podStartE2EDuration="11.032880678s" totalImagesPullingTime="8.077552863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:27.032451376 +0000 UTC m=+11.838507248" watchObservedRunningTime="2026-11-03 19:56:27.032880678 +0000 UTC m=+11.838936548" Nov 03 19:56:27.067216 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:27.067158 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-ksm7q" podStartSLOduration=2.650817828 podStartE2EDuration="11.067145539s" totalImagesPullingTime="8.416327711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:27.067015338 +0000 UTC m=+11.873071211" watchObservedRunningTime="2026-11-03 19:56:27.067145539 +0000 UTC m=+11.873201409" Nov 03 19:56:27.837702 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:27.837677 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:27.838173 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:27.837739 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:27.838173 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:27.837839 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvsdz" podUID="d2e4493e-d29f-49b6-851e-cbd01214bf5c" Nov 03 19:56:27.838173 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:27.837736 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:27.838173 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:27.837939 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dcqpv" podUID="ab34f1da-ace1-443d-bddd-9d9905d22e52" Nov 03 19:56:27.838173 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:27.838035 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2ms" podUID="f90a0a4e-2794-46fc-a799-9c1593d859ab" Nov 03 19:56:27.994555 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:27.994518 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fjr5q" event={"ID":"a24da6cd-d805-4f8a-ac99-89dbd45c86c3","Type":"ContainerStarted","Data":"194ac34a36119e62499bb6b75d22c2fa7858ea7d32607264ffcf8bdc6be75ca1"} Nov 03 19:56:27.998675 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:27.998644 2581 generic.go:356] "Generic (PLEG): container finished" podID="88c8909b-9acd-46d0-9f19-d74e2db51f77" containerID="08632f5eea60d7461fb62d450fbc7f7b62ecc39ec0e958b9fcc14d8c9ce16398" exitCode=0 Nov 03 19:56:27.998801 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:27.998741 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjgg6" event={"ID":"88c8909b-9acd-46d0-9f19-d74e2db51f77","Type":"ContainerDied","Data":"08632f5eea60d7461fb62d450fbc7f7b62ecc39ec0e958b9fcc14d8c9ce16398"} Nov 03 19:56:28.036352 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:28.036295 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fjr5q" podStartSLOduration=3.984857812 podStartE2EDuration="12.036281974s" totalImagesPullingTime="8.051424162s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:28.011465652 +0000 UTC m=+12.817521522" watchObservedRunningTime="2026-11-03 19:56:28.036281974 +0000 UTC m=+12.842337844" Nov 03 19:56:29.002800 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:29.002765 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" event={"ID":"f838cffc-2fc7-4a3f-880b-8a80298cb1b4","Type":"ContainerStarted","Data":"7f6835bd76ce65b41e3d1ee321401ada116644a3c395556e27e03194e90162a5"} Nov 03 19:56:29.023896 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:29.023849 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" podStartSLOduration=2.482762153 podStartE2EDuration="13.023831464s" totalImagesPullingTime="10.541069311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:29.02322364 +0000 UTC m=+13.829279511" watchObservedRunningTime="2026-11-03 19:56:29.023831464 +0000 UTC m=+13.829887338" Nov 03 19:56:29.838024 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:29.837984 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:29.838024 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:29.837984 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:29.838265 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:29.838147 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvsdz" podUID="d2e4493e-d29f-49b6-851e-cbd01214bf5c" Nov 03 19:56:29.838504 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:29.838487 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:29.838596 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:29.838577 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2ms" podUID="f90a0a4e-2794-46fc-a799-9c1593d859ab" Nov 03 19:56:29.838685 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:29.838668 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dcqpv" podUID="ab34f1da-ace1-443d-bddd-9d9905d22e52" Nov 03 19:56:30.007732 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:30.007697 2581 generic.go:356] "Generic (PLEG): container finished" podID="88c8909b-9acd-46d0-9f19-d74e2db51f77" containerID="a38de7e690d459b8b35791ed408515ab1910b5cfd4405a986c8f2c5a04ab93f7" exitCode=0 Nov 03 19:56:30.008283 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:30.007787 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjgg6" event={"ID":"88c8909b-9acd-46d0-9f19-d74e2db51f77","Type":"ContainerDied","Data":"a38de7e690d459b8b35791ed408515ab1910b5cfd4405a986c8f2c5a04ab93f7"} Nov 03 19:56:31.838123 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:31.837871 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:31.838599 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:31.837905 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:31.838599 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:31.838250 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvsdz" podUID="d2e4493e-d29f-49b6-851e-cbd01214bf5c" Nov 03 19:56:31.838599 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:31.837949 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:31.838599 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:31.838316 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2ms" podUID="f90a0a4e-2794-46fc-a799-9c1593d859ab" Nov 03 19:56:31.838599 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:31.838372 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dcqpv" podUID="ab34f1da-ace1-443d-bddd-9d9905d22e52" Nov 03 19:56:32.015573 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:32.015455 2581 generic.go:356] "Generic (PLEG): container finished" podID="88c8909b-9acd-46d0-9f19-d74e2db51f77" containerID="10a92b2b943ae59e4711069fc7cd1fe05a72f072578820e08714063a9bcc0636" exitCode=0 Nov 03 19:56:32.015573 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:32.015525 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjgg6" event={"ID":"88c8909b-9acd-46d0-9f19-d74e2db51f77","Type":"ContainerDied","Data":"10a92b2b943ae59e4711069fc7cd1fe05a72f072578820e08714063a9bcc0636"} Nov 03 19:56:33.024085 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:33.023921 2581 generic.go:356] "Generic (PLEG): container finished" podID="88c8909b-9acd-46d0-9f19-d74e2db51f77" containerID="396c87eeab78084d270a553e6d86c9263cdacbf1f6382b8183d7db97fa2bd9c3" exitCode=0 Nov 03 19:56:33.024581 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:33.024007 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjgg6" event={"ID":"88c8909b-9acd-46d0-9f19-d74e2db51f77","Type":"ContainerDied","Data":"396c87eeab78084d270a553e6d86c9263cdacbf1f6382b8183d7db97fa2bd9c3"} Nov 03 19:56:33.027766 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:33.027746 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" event={"ID":"99713fae-ae74-4da5-b435-c73edc1c4843","Type":"ContainerStarted","Data":"13c588608aec94e23e003021896e9226dca693b882b5d2ad4ff6702d2d1dec8f"} Nov 03 19:56:33.027847 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:33.027799 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" event={"ID":"99713fae-ae74-4da5-b435-c73edc1c4843","Type":"ContainerStarted","Data":"f42177b213e5f5af4d16148308bfa983071b9e82964f85d1c54dd3abfbdf0096"} Nov 03 19:56:33.027847 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:33.027829 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" event={"ID":"99713fae-ae74-4da5-b435-c73edc1c4843","Type":"ContainerStarted","Data":"99cd3fbcad6eb5b98eb1b1aaf24e3c398119fba7cc4c4a484122f8f679517e66"} Nov 03 19:56:33.027905 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:33.027856 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" event={"ID":"99713fae-ae74-4da5-b435-c73edc1c4843","Type":"ContainerStarted","Data":"c23967b603549a2cf5ec5d7ca39e17fa6a6da7b0d8b4869950b5fac97e5a4654"} Nov 03 19:56:33.027905 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:33.027882 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" event={"ID":"99713fae-ae74-4da5-b435-c73edc1c4843","Type":"ContainerStarted","Data":"317ec4f585a93686e4ac4a03cebe8f4fb9af9d6601b1a31ea8be541d36addf4b"} Nov 03 19:56:33.358078 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:33.358046 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:33.358257 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:33.358087 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs\") pod \"network-metrics-daemon-dcqpv\" (UID: \"ab34f1da-ace1-443d-bddd-9d9905d22e52\") " pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:33.358257 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:33.358209 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:33.358336 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:33.358269 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret podName:d2e4493e-d29f-49b6-851e-cbd01214bf5c nodeName:}" failed. No retries permitted until 2026-11-03 19:56:49.358255935 +0000 UTC m=+34.164311787 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret") pod "global-pull-secret-syncer-mvsdz" (UID: "d2e4493e-d29f-49b6-851e-cbd01214bf5c") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:33.358336 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:33.358209 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:33.358336 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:33.358309 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs podName:ab34f1da-ace1-443d-bddd-9d9905d22e52 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:49.358299225 +0000 UTC m=+34.164355075 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs") pod "network-metrics-daemon-dcqpv" (UID: "ab34f1da-ace1-443d-bddd-9d9905d22e52") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:33.458467 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:33.458433 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsb75\" (UniqueName: \"kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75\") pod \"network-check-target-zn2ms\" (UID: \"f90a0a4e-2794-46fc-a799-9c1593d859ab\") " pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:33.458640 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:33.458619 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:56:33.458699 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:33.458647 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:56:33.458699 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:33.458662 2581 projected.go:196] Error preparing data for projected volume kube-api-access-tsb75 for pod openshift-network-diagnostics/network-check-target-zn2ms: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:33.458785 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:33.458732 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75 podName:f90a0a4e-2794-46fc-a799-9c1593d859ab nodeName:}" failed. No retries permitted until 2026-11-03 19:56:49.458707641 +0000 UTC m=+34.264763511 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tsb75" (UniqueName: "kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75") pod "network-check-target-zn2ms" (UID: "f90a0a4e-2794-46fc-a799-9c1593d859ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:33.837944 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:33.837919 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:33.837944 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:33.837938 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:33.838104 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:33.837951 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:33.838104 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:33.838056 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2ms" podUID="f90a0a4e-2794-46fc-a799-9c1593d859ab" Nov 03 19:56:33.838215 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:33.838173 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dcqpv" podUID="ab34f1da-ace1-443d-bddd-9d9905d22e52" Nov 03 19:56:33.838327 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:33.838262 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvsdz" podUID="d2e4493e-d29f-49b6-851e-cbd01214bf5c" Nov 03 19:56:34.033153 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:34.033117 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjgg6" event={"ID":"88c8909b-9acd-46d0-9f19-d74e2db51f77","Type":"ContainerStarted","Data":"44c24cae2cbfd6ab3346948e1cceee17917645760db89441d52e842a5c80a992"} Nov 03 19:56:34.036008 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:34.035984 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" event={"ID":"99713fae-ae74-4da5-b435-c73edc1c4843","Type":"ContainerStarted","Data":"b535f62fbe5f0353cf58d11d6960660aa1081683971746837771f62c492eb569"} Nov 03 19:56:34.060109 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:34.060073 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wjgg6" podStartSLOduration=7.33007302 podStartE2EDuration="18.060064273s" totalImagesPullingTime="10.729991253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:34.05845562 +0000 UTC m=+18.864511490" watchObservedRunningTime="2026-11-03 19:56:34.060064273 +0000 UTC m=+18.866120142" Nov 03 19:56:35.838864 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:35.838835 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:35.839343 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:35.838911 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvsdz" podUID="d2e4493e-d29f-49b6-851e-cbd01214bf5c" Nov 03 19:56:35.839343 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:35.839010 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:35.839343 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:35.839116 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2ms" podUID="f90a0a4e-2794-46fc-a799-9c1593d859ab" Nov 03 19:56:35.839343 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:35.839195 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:35.839343 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:35.839284 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dcqpv" podUID="ab34f1da-ace1-443d-bddd-9d9905d22e52" Nov 03 19:56:36.043051 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:36.043015 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" event={"ID":"99713fae-ae74-4da5-b435-c73edc1c4843","Type":"ContainerStarted","Data":"148a4f34f7659c89b51ca22575f30e1f34ebe1bcbba6de649dd5aec0f089cc44"} Nov 03 19:56:37.837773 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:37.837589 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:37.838271 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:37.837654 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:37.838271 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:37.837859 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dcqpv" podUID="ab34f1da-ace1-443d-bddd-9d9905d22e52" Nov 03 19:56:37.838271 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:37.837678 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:37.838271 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:37.837926 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2ms" podUID="f90a0a4e-2794-46fc-a799-9c1593d859ab" Nov 03 19:56:37.838271 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:37.837992 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvsdz" podUID="d2e4493e-d29f-49b6-851e-cbd01214bf5c" Nov 03 19:56:38.050327 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:38.050293 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" event={"ID":"99713fae-ae74-4da5-b435-c73edc1c4843","Type":"ContainerStarted","Data":"47cc4e1cade4e484baf4a58e0cefc3cf9d3d95c2ada94ef6044b55ad7c878595"} Nov 03 19:56:38.050548 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:38.050534 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:38.050619 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:38.050579 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:38.067129 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:38.067107 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:38.118403 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:38.118326 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" podStartSLOduration=7.444490336 podStartE2EDuration="22.118313708s" totalImagesPullingTime="14.673823372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:38.08546158 +0000 UTC m=+22.891517448" watchObservedRunningTime="2026-11-03 19:56:38.118313708 +0000 UTC m=+22.924369580" Nov 03 19:56:38.899394 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:38.899365 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mvsdz"] Nov 03 19:56:38.899787 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:38.899513 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:38.899787 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:38.899600 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvsdz" podUID="d2e4493e-d29f-49b6-851e-cbd01214bf5c" Nov 03 19:56:38.902336 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:38.902314 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zn2ms"] Nov 03 19:56:38.902441 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:38.902392 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:38.902477 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:38.902455 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2ms" podUID="f90a0a4e-2794-46fc-a799-9c1593d859ab" Nov 03 19:56:38.905216 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:38.905194 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dcqpv"] Nov 03 19:56:38.905299 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:38.905282 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:38.905371 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:38.905356 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dcqpv" podUID="ab34f1da-ace1-443d-bddd-9d9905d22e52" Nov 03 19:56:39.053404 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:39.053380 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:39.067932 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:39.067908 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:40.838192 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:40.838146 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:40.838666 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:40.838280 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:40.838666 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:40.838284 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvsdz" podUID="d2e4493e-d29f-49b6-851e-cbd01214bf5c" Nov 03 19:56:40.838666 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:40.838362 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2ms" podUID="f90a0a4e-2794-46fc-a799-9c1593d859ab" Nov 03 19:56:40.838666 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:40.838385 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:40.838666 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:40.838448 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dcqpv" podUID="ab34f1da-ace1-443d-bddd-9d9905d22e52" Nov 03 19:56:42.837901 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:42.837865 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:42.838472 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:42.837940 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:42.838472 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:42.838020 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvsdz" podUID="d2e4493e-d29f-49b6-851e-cbd01214bf5c" Nov 03 19:56:42.838472 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:42.838046 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:42.838472 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:42.838111 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dcqpv" podUID="ab34f1da-ace1-443d-bddd-9d9905d22e52" Nov 03 19:56:42.838472 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:42.838191 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2ms" podUID="f90a0a4e-2794-46fc-a799-9c1593d859ab" Nov 03 19:56:44.838200 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:44.838149 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:44.838200 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:44.838189 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:44.838733 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:44.838273 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:44.838733 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:44.838295 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvsdz" podUID="d2e4493e-d29f-49b6-851e-cbd01214bf5c" Nov 03 19:56:44.838733 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:44.838348 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2ms" podUID="f90a0a4e-2794-46fc-a799-9c1593d859ab" Nov 03 19:56:44.838733 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:44.838418 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dcqpv" podUID="ab34f1da-ace1-443d-bddd-9d9905d22e52" Nov 03 19:56:45.460192 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.459998 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeReady" Nov 03 19:56:45.460345 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.460298 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 19:56:45.495639 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.495573 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6r7g9"] Nov 03 19:56:45.497959 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.497944 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6r7g9" Nov 03 19:56:45.501468 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.501449 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qkzrh"] Nov 03 19:56:45.509390 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.508708 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2zqfd"] Nov 03 19:56:45.510073 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.510015 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qkzrh" Nov 03 19:56:45.513826 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.513793 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 19:56:45.513826 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.513803 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 19:56:45.514746 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.514726 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jk8h6"] Nov 03 19:56:45.514847 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.514809 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:56:45.518037 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.518021 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st"] Nov 03 19:56:45.518445 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.518167 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:45.520851 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.520836 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zl889"] Nov 03 19:56:45.520948 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.520936 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:45.523915 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.523899 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5dgdg"] Nov 03 19:56:45.524027 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.524011 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:45.526833 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.526771 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq"] Nov 03 19:56:45.527784 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.527660 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5dgdg" Nov 03 19:56:45.532694 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.532668 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 19:56:45.532794 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.532775 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-xd6ln\"" Nov 03 19:56:45.532875 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.532860 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 19:56:45.532935 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.532904 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 19:56:45.532990 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.532966 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ft7x5\"" Nov 03 19:56:45.533063 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.533049 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6mlp2\"" Nov 03 19:56:45.533121 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.533077 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 19:56:45.533201 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.533166 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dfmhm\"" Nov 03 19:56:45.534751 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.534722 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-9d99c5c47-ttssq"] Nov 03 19:56:45.534937 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.534919 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" Nov 03 19:56:45.538338 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.538319 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mvxwm"] Nov 03 19:56:45.538446 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.538430 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.541052 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.541037 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8"] Nov 03 19:56:45.541168 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.541152 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mvxwm" Nov 03 19:56:45.543659 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.543639 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d95b6bf4b-dd7kz"] Nov 03 19:56:45.543882 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.543842 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:45.557619 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.557596 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4"] Nov 03 19:56:45.557889 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.557867 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.560635 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.560610 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6bfcd89b85-9qb4s"] Nov 03 19:56:45.560804 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.560768 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" Nov 03 19:56:45.563395 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.563379 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-qtbqg"] Nov 03 19:56:45.563511 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.563498 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:45.566401 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.566379 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6r7g9"] Nov 03 19:56:45.566527 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.566416 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2zqfd"] Nov 03 19:56:45.566527 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.566441 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-95k62"] Nov 03 19:56:45.566754 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.566685 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qtbqg" Nov 03 19:56:45.569354 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.569336 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq"] Nov 03 19:56:45.569428 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.569374 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st"] Nov 03 19:56:45.569428 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.569397 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mvxwm"] Nov 03 19:56:45.569428 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.569419 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xsh7l"] Nov 03 19:56:45.569554 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.569433 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-95k62" Nov 03 19:56:45.572556 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.572535 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qkzrh"] Nov 03 19:56:45.572634 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.572568 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5dgdg"] Nov 03 19:56:45.572634 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.572589 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8"] Nov 03 19:56:45.572634 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.572615 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-qtbqg"] Nov 03 19:56:45.572738 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.572641 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6bfcd89b85-9qb4s"] Nov 03 19:56:45.572738 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.572643 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:45.572738 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.572662 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jk8h6"] Nov 03 19:56:45.572738 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.572683 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-95k62"] Nov 03 19:56:45.572738 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.572699 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4"] Nov 03 19:56:45.572738 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.572711 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zl889"] Nov 03 19:56:45.572738 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.572721 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xsh7l"] Nov 03 19:56:45.573003 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.572742 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9d99c5c47-ttssq"] Nov 03 19:56:45.573003 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.572763 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d95b6bf4b-dd7kz"] Nov 03 19:56:45.605729 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.605706 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 19:56:45.605831 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.605763 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:56:45.605831 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.605764 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 19:56:45.606775 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.606749 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 19:56:45.606775 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.606770 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:56:45.606923 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.606782 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 19:56:45.606923 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.606794 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 19:56:45.606923 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.606780 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:56:45.606923 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.606836 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 19:56:45.606923 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.606869 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 19:56:45.606923 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.606875 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 19:56:45.606923 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.606904 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 19:56:45.606923 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.606904 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 19:56:45.606923 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.606915 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 19:56:45.607265 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.606936 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 19:56:45.607265 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.606954 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 19:56:45.607265 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.606973 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 19:56:45.607265 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.606986 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 19:56:45.607265 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.607022 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 19:56:45.612298 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.612282 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 19:56:45.612414 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.612332 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:56:45.612414 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.612337 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 19:56:45.612942 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.612917 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 19:56:45.613018 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.612964 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:56:45.613018 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.613008 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 19:56:45.613240 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.613224 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 19:56:45.613383 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.613367 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 19:56:45.613526 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.613512 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 19:56:45.613620 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.613604 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 19:56:45.622448 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.622414 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 19:56:45.622448 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.622437 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 19:56:45.624312 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.624257 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 19:56:45.630301 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.628934 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 19:56:45.630301 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.629000 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 19:56:45.630301 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.629042 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nshbb\"" Nov 03 19:56:45.630301 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.629108 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 19:56:45.630301 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.629231 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 19:56:45.630301 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.629316 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 19:56:45.630301 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.629319 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 19:56:45.630301 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.629347 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 19:56:45.630301 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.629379 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 19:56:45.630301 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.629417 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 19:56:45.630301 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.629471 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8rzcq\"" Nov 03 19:56:45.630301 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.629541 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qbp66\"" Nov 03 19:56:45.630301 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.629609 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 19:56:45.630301 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.629694 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9lnsw\"" Nov 03 19:56:45.630301 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.630309 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jwsg8\"" Nov 03 19:56:45.630828 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.630508 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 19:56:45.630828 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.630520 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-f6mkl\"" Nov 03 19:56:45.630828 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.630549 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rpk57\"" Nov 03 19:56:45.630828 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.630571 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5rc2l\"" Nov 03 19:56:45.630828 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.630584 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4mwrd\"" Nov 03 19:56:45.644781 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.644762 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/449a05ee-ead2-4507-8d65-faaeb3ab3281-service-ca-bundle\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:45.644861 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.644788 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml5nd\" (UniqueName: \"kubernetes.io/projected/4726eb4c-48e8-423d-a62c-250a18bd8b83-kube-api-access-ml5nd\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:45.644861 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.644809 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5sh5\" (UniqueName: \"kubernetes.io/projected/aed0ee7d-d648-4138-b5ef-2ae6795321d2-kube-api-access-w5sh5\") pod \"ingress-canary-95k62\" (UID: \"aed0ee7d-d648-4138-b5ef-2ae6795321d2\") " pod="openshift-ingress-canary/ingress-canary-95k62" Nov 03 19:56:45.644861 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.644857 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2zqfd\" (UID: \"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:56:45.644957 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.644899 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44-serving-cert\") pod \"service-ca-operator-544d749c8c-9ncj4\" (UID: \"23f64ae9-fdda-4f2e-b3a5-1c552cc55a44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" Nov 03 19:56:45.644989 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.644975 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/449a05ee-ead2-4507-8d65-faaeb3ab3281-snapshots\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:45.645050 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645033 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/adc4cee9-00f0-4fac-b655-f58c21377362-image-registry-private-configuration\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.645083 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645063 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ff051e4a-95a6-4c0d-9f69-773974d6eb7c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq\" (UID: \"ff051e4a-95a6-4c0d-9f69-773974d6eb7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" Nov 03 19:56:45.645115 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645092 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-certificates\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.645146 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645114 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtdbf\" (UniqueName: \"kubernetes.io/projected/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-kube-api-access-gtdbf\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:45.645146 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645142 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4726eb4c-48e8-423d-a62c-250a18bd8b83-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:45.645218 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645170 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/adc4cee9-00f0-4fac-b655-f58c21377362-registry-certificates\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.645255 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645229 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f594\" (UniqueName: \"kubernetes.io/projected/6de4fd5d-d10a-4630-80bf-3838fb02d679-kube-api-access-8f594\") pod \"network-check-source-6f78d59f99-qtbqg\" (UID: \"6de4fd5d-d10a-4630-80bf-3838fb02d679\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qtbqg" Nov 03 19:56:45.645286 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645260 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff051e4a-95a6-4c0d-9f69-773974d6eb7c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq\" (UID: \"ff051e4a-95a6-4c0d-9f69-773974d6eb7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" Nov 03 19:56:45.645321 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645282 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb6j8\" (UniqueName: \"kubernetes.io/projected/653cb377-919b-4793-85a4-0d60d6aeaff6-kube-api-access-vb6j8\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:45.645321 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645315 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x4lc\" (UniqueName: \"kubernetes.io/projected/433c5b6b-176b-44d1-afc0-6884045e468c-kube-api-access-8x4lc\") pod \"console-operator-7cd56b8dbc-jk8h6\" (UID: \"433c5b6b-176b-44d1-afc0-6884045e468c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:45.645375 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645337 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-stats-auth\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:45.645375 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645362 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/514e4bfb-77ef-4b1f-977a-3f9308cf886c-installation-pull-secrets\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.645437 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645389 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hb8f\" (UniqueName: \"kubernetes.io/projected/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44-kube-api-access-6hb8f\") pod \"service-ca-operator-544d749c8c-9ncj4\" (UID: \"23f64ae9-fdda-4f2e-b3a5-1c552cc55a44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" Nov 03 19:56:45.645437 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645404 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aed0ee7d-d648-4138-b5ef-2ae6795321d2-cert\") pod \"ingress-canary-95k62\" (UID: \"aed0ee7d-d648-4138-b5ef-2ae6795321d2\") " pod="openshift-ingress-canary/ingress-canary-95k62" Nov 03 19:56:45.645437 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645433 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/449a05ee-ead2-4507-8d65-faaeb3ab3281-tmp\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:45.645520 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645464 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:45.645520 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645514 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.645580 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645536 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2zqfd\" (UID: \"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:56:45.645649 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645636 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/449a05ee-ead2-4507-8d65-faaeb3ab3281-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:45.645681 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645657 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm2tl\" (UniqueName: \"kubernetes.io/projected/ff051e4a-95a6-4c0d-9f69-773974d6eb7c-kube-api-access-gm2tl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq\" (UID: \"ff051e4a-95a6-4c0d-9f69-773974d6eb7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" Nov 03 19:56:45.645681 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645675 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44-config\") pod \"service-ca-operator-544d749c8c-9ncj4\" (UID: \"23f64ae9-fdda-4f2e-b3a5-1c552cc55a44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" Nov 03 19:56:45.645739 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645692 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd49x\" (UniqueName: \"kubernetes.io/projected/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-kube-api-access-dd49x\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:45.645739 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645709 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/adc4cee9-00f0-4fac-b655-f58c21377362-ca-trust-extracted\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.645792 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645743 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:45.645792 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645778 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/433c5b6b-176b-44d1-afc0-6884045e468c-serving-cert\") pod \"console-operator-7cd56b8dbc-jk8h6\" (UID: \"433c5b6b-176b-44d1-afc0-6884045e468c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:45.645853 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645810 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-default-certificate\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:45.645853 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645844 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-bound-sa-token\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.645910 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645868 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:45.645910 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645900 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/514e4bfb-77ef-4b1f-977a-3f9308cf886c-image-registry-private-configuration\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.645969 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645933 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sx9r\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-kube-api-access-6sx9r\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.645969 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645952 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:45.646026 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645977 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/514e4bfb-77ef-4b1f-977a-3f9308cf886c-ca-trust-extracted\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.646026 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.645992 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adc4cee9-00f0-4fac-b655-f58c21377362-trusted-ca\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.646082 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.646028 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:45.646082 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.646047 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxkg4\" (UniqueName: \"kubernetes.io/projected/449a05ee-ead2-4507-8d65-faaeb3ab3281-kube-api-access-zxkg4\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:45.646082 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.646072 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.646164 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.646096 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/adc4cee9-00f0-4fac-b655-f58c21377362-installation-pull-secrets\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.646164 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.646115 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-config-volume\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:45.646164 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.646136 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vgq9\" (UniqueName: \"kubernetes.io/projected/c60cb950-5dd9-4a2b-bba3-b3c38d3591e3-kube-api-access-6vgq9\") pod \"volume-data-source-validator-58c9f5f46d-mvxwm\" (UID: \"c60cb950-5dd9-4a2b-bba3-b3c38d3591e3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mvxwm" Nov 03 19:56:45.646164 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.646156 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djg6c\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-kube-api-access-djg6c\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.646283 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.646196 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl89h\" (UniqueName: \"kubernetes.io/projected/158d8c01-2d31-43b4-a098-0c731c1fd163-kube-api-access-wl89h\") pod \"kas-connection-checker-685895c7c9-6r7g9\" (UID: \"158d8c01-2d31-43b4-a098-0c731c1fd163\") " pod="kube-system/kas-connection-checker-685895c7c9-6r7g9" Nov 03 19:56:45.646283 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.646234 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/514e4bfb-77ef-4b1f-977a-3f9308cf886c-trusted-ca\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.646283 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.646263 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-tmp-dir\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:45.646370 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.646303 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/433c5b6b-176b-44d1-afc0-6884045e468c-config\") pod \"console-operator-7cd56b8dbc-jk8h6\" (UID: \"433c5b6b-176b-44d1-afc0-6884045e468c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:45.646370 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.646329 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkf49\" (UniqueName: \"kubernetes.io/projected/5a7d187d-9a17-475b-8ee4-ae5836d9a73a-kube-api-access-gkf49\") pod \"kas-connection-checker-685895c7c9-qkzrh\" (UID: \"5a7d187d-9a17-475b-8ee4-ae5836d9a73a\") " pod="kube-system/kas-connection-checker-685895c7c9-qkzrh" Nov 03 19:56:45.646370 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.646355 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/449a05ee-ead2-4507-8d65-faaeb3ab3281-serving-cert\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:45.646454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.646373 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/433c5b6b-176b-44d1-afc0-6884045e468c-trusted-ca\") pod \"console-operator-7cd56b8dbc-jk8h6\" (UID: \"433c5b6b-176b-44d1-afc0-6884045e468c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:45.646454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.646393 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:45.646454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.646408 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh8xj\" (UniqueName: \"kubernetes.io/projected/61f1d7a9-a8b3-4db5-878d-4b8d1632fafc-kube-api-access-jh8xj\") pod \"kas-connection-checker-685895c7c9-5dgdg\" (UID: \"61f1d7a9-a8b3-4db5-878d-4b8d1632fafc\") " pod="kube-system/kas-connection-checker-685895c7c9-5dgdg" Nov 03 19:56:45.646454 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.646424 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-bound-sa-token\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.747262 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747189 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh8xj\" (UniqueName: \"kubernetes.io/projected/61f1d7a9-a8b3-4db5-878d-4b8d1632fafc-kube-api-access-jh8xj\") pod \"kas-connection-checker-685895c7c9-5dgdg\" (UID: \"61f1d7a9-a8b3-4db5-878d-4b8d1632fafc\") " pod="kube-system/kas-connection-checker-685895c7c9-5dgdg" Nov 03 19:56:45.747262 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747220 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-bound-sa-token\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.747262 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747236 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/449a05ee-ead2-4507-8d65-faaeb3ab3281-service-ca-bundle\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:45.747262 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747254 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml5nd\" (UniqueName: \"kubernetes.io/projected/4726eb4c-48e8-423d-a62c-250a18bd8b83-kube-api-access-ml5nd\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:45.747262 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747268 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5sh5\" (UniqueName: \"kubernetes.io/projected/aed0ee7d-d648-4138-b5ef-2ae6795321d2-kube-api-access-w5sh5\") pod \"ingress-canary-95k62\" (UID: \"aed0ee7d-d648-4138-b5ef-2ae6795321d2\") " pod="openshift-ingress-canary/ingress-canary-95k62" Nov 03 19:56:45.747585 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747455 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2zqfd\" (UID: \"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:56:45.747585 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747484 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44-serving-cert\") pod \"service-ca-operator-544d749c8c-9ncj4\" (UID: \"23f64ae9-fdda-4f2e-b3a5-1c552cc55a44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" Nov 03 19:56:45.747585 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747507 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/449a05ee-ead2-4507-8d65-faaeb3ab3281-snapshots\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:45.747585 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747531 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/adc4cee9-00f0-4fac-b655-f58c21377362-image-registry-private-configuration\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.747777 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747586 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ff051e4a-95a6-4c0d-9f69-773974d6eb7c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq\" (UID: \"ff051e4a-95a6-4c0d-9f69-773974d6eb7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" Nov 03 19:56:45.747777 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747628 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-certificates\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.747777 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747657 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtdbf\" (UniqueName: \"kubernetes.io/projected/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-kube-api-access-gtdbf\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:45.747777 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747689 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4726eb4c-48e8-423d-a62c-250a18bd8b83-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:45.747777 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747717 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/adc4cee9-00f0-4fac-b655-f58c21377362-registry-certificates\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.747777 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747743 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f594\" (UniqueName: \"kubernetes.io/projected/6de4fd5d-d10a-4630-80bf-3838fb02d679-kube-api-access-8f594\") pod \"network-check-source-6f78d59f99-qtbqg\" (UID: \"6de4fd5d-d10a-4630-80bf-3838fb02d679\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qtbqg" Nov 03 19:56:45.747777 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747775 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff051e4a-95a6-4c0d-9f69-773974d6eb7c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq\" (UID: \"ff051e4a-95a6-4c0d-9f69-773974d6eb7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" Nov 03 19:56:45.748090 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747801 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb6j8\" (UniqueName: \"kubernetes.io/projected/653cb377-919b-4793-85a4-0d60d6aeaff6-kube-api-access-vb6j8\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:45.748090 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747829 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x4lc\" (UniqueName: \"kubernetes.io/projected/433c5b6b-176b-44d1-afc0-6884045e468c-kube-api-access-8x4lc\") pod \"console-operator-7cd56b8dbc-jk8h6\" (UID: \"433c5b6b-176b-44d1-afc0-6884045e468c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:45.748090 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747855 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-stats-auth\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:45.748090 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747882 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/514e4bfb-77ef-4b1f-977a-3f9308cf886c-installation-pull-secrets\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.748090 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747908 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hb8f\" (UniqueName: \"kubernetes.io/projected/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44-kube-api-access-6hb8f\") pod \"service-ca-operator-544d749c8c-9ncj4\" (UID: \"23f64ae9-fdda-4f2e-b3a5-1c552cc55a44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" Nov 03 19:56:45.748090 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747926 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/449a05ee-ead2-4507-8d65-faaeb3ab3281-service-ca-bundle\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:45.748090 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747932 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aed0ee7d-d648-4138-b5ef-2ae6795321d2-cert\") pod \"ingress-canary-95k62\" (UID: \"aed0ee7d-d648-4138-b5ef-2ae6795321d2\") " pod="openshift-ingress-canary/ingress-canary-95k62" Nov 03 19:56:45.748090 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747961 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/449a05ee-ead2-4507-8d65-faaeb3ab3281-tmp\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:45.748090 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.747989 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:45.748090 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748021 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.748090 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748047 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2zqfd\" (UID: \"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:56:45.748618 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748098 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/449a05ee-ead2-4507-8d65-faaeb3ab3281-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:45.748618 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748128 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm2tl\" (UniqueName: \"kubernetes.io/projected/ff051e4a-95a6-4c0d-9f69-773974d6eb7c-kube-api-access-gm2tl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq\" (UID: \"ff051e4a-95a6-4c0d-9f69-773974d6eb7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" Nov 03 19:56:45.748618 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748156 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/449a05ee-ead2-4507-8d65-faaeb3ab3281-snapshots\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:45.748618 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748173 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44-config\") pod \"service-ca-operator-544d749c8c-9ncj4\" (UID: \"23f64ae9-fdda-4f2e-b3a5-1c552cc55a44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" Nov 03 19:56:45.748618 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748221 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd49x\" (UniqueName: \"kubernetes.io/projected/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-kube-api-access-dd49x\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:45.748618 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748253 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/adc4cee9-00f0-4fac-b655-f58c21377362-ca-trust-extracted\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.748618 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748278 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:45.748618 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:45.748303 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 19:56:45.748618 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748315 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/433c5b6b-176b-44d1-afc0-6884045e468c-serving-cert\") pod \"console-operator-7cd56b8dbc-jk8h6\" (UID: \"433c5b6b-176b-44d1-afc0-6884045e468c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:45.748618 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748343 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-default-certificate\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:45.748618 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:45.748380 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls podName:4726eb4c-48e8-423d-a62c-250a18bd8b83 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:46.248358197 +0000 UTC m=+31.054414063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nb7st" (UID: "4726eb4c-48e8-423d-a62c-250a18bd8b83") : secret "cluster-monitoring-operator-tls" not found Nov 03 19:56:45.748618 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748404 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-bound-sa-token\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.748618 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748437 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2zqfd\" (UID: \"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:56:45.748618 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748438 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:45.748618 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748468 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-certificates\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.748618 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:45.748501 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:56:45.749269 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:45.748538 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls podName:93a3cf28-8c0c-4735-b177-d28bc7c8ea3f nodeName:}" failed. No retries permitted until 2026-11-03 19:56:46.248526018 +0000 UTC m=+31.054581879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls") pod "dns-default-xsh7l" (UID: "93a3cf28-8c0c-4735-b177-d28bc7c8ea3f") : secret "dns-default-metrics-tls" not found Nov 03 19:56:45.749269 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748501 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/514e4bfb-77ef-4b1f-977a-3f9308cf886c-image-registry-private-configuration\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.749269 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748573 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sx9r\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-kube-api-access-6sx9r\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.749269 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748603 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:45.749269 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748642 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/514e4bfb-77ef-4b1f-977a-3f9308cf886c-ca-trust-extracted\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.749269 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748668 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adc4cee9-00f0-4fac-b655-f58c21377362-trusted-ca\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.749269 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748697 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:45.749269 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748723 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxkg4\" (UniqueName: \"kubernetes.io/projected/449a05ee-ead2-4507-8d65-faaeb3ab3281-kube-api-access-zxkg4\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:45.749269 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748748 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.749269 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748772 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/adc4cee9-00f0-4fac-b655-f58c21377362-installation-pull-secrets\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.749269 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748798 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-config-volume\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:45.749269 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748827 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vgq9\" (UniqueName: \"kubernetes.io/projected/c60cb950-5dd9-4a2b-bba3-b3c38d3591e3-kube-api-access-6vgq9\") pod \"volume-data-source-validator-58c9f5f46d-mvxwm\" (UID: \"c60cb950-5dd9-4a2b-bba3-b3c38d3591e3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mvxwm" Nov 03 19:56:45.749269 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748854 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djg6c\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-kube-api-access-djg6c\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.749269 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748880 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl89h\" (UniqueName: \"kubernetes.io/projected/158d8c01-2d31-43b4-a098-0c731c1fd163-kube-api-access-wl89h\") pod \"kas-connection-checker-685895c7c9-6r7g9\" (UID: \"158d8c01-2d31-43b4-a098-0c731c1fd163\") " pod="kube-system/kas-connection-checker-685895c7c9-6r7g9" Nov 03 19:56:45.749269 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748910 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/514e4bfb-77ef-4b1f-977a-3f9308cf886c-trusted-ca\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.749924 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748934 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-tmp-dir\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:45.749924 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748962 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/433c5b6b-176b-44d1-afc0-6884045e468c-config\") pod \"console-operator-7cd56b8dbc-jk8h6\" (UID: \"433c5b6b-176b-44d1-afc0-6884045e468c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:45.749924 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.748989 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkf49\" (UniqueName: \"kubernetes.io/projected/5a7d187d-9a17-475b-8ee4-ae5836d9a73a-kube-api-access-gkf49\") pod \"kas-connection-checker-685895c7c9-qkzrh\" (UID: \"5a7d187d-9a17-475b-8ee4-ae5836d9a73a\") " pod="kube-system/kas-connection-checker-685895c7c9-qkzrh" Nov 03 19:56:45.749924 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.749031 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/449a05ee-ead2-4507-8d65-faaeb3ab3281-serving-cert\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:45.749924 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.749066 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/433c5b6b-176b-44d1-afc0-6884045e468c-trusted-ca\") pod \"console-operator-7cd56b8dbc-jk8h6\" (UID: \"433c5b6b-176b-44d1-afc0-6884045e468c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:45.749924 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.749093 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:45.749924 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.749102 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4726eb4c-48e8-423d-a62c-250a18bd8b83-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:45.749924 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.749149 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff051e4a-95a6-4c0d-9f69-773974d6eb7c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq\" (UID: \"ff051e4a-95a6-4c0d-9f69-773974d6eb7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" Nov 03 19:56:45.749924 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:45.749241 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:56:45.749924 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:45.749255 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9d99c5c47-ttssq: secret "image-registry-tls" not found Nov 03 19:56:45.749924 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:45.749294 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls podName:adc4cee9-00f0-4fac-b655-f58c21377362 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:46.249281632 +0000 UTC m=+31.055337493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls") pod "image-registry-9d99c5c47-ttssq" (UID: "adc4cee9-00f0-4fac-b655-f58c21377362") : secret "image-registry-tls" not found Nov 03 19:56:45.749924 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:45.749495 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:56:45.749924 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:45.749510 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d95b6bf4b-dd7kz: secret "image-registry-tls" not found Nov 03 19:56:45.749924 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:45.749547 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls podName:514e4bfb-77ef-4b1f-977a-3f9308cf886c nodeName:}" failed. No retries permitted until 2026-11-03 19:56:46.249535471 +0000 UTC m=+31.055591337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls") pod "image-registry-5d95b6bf4b-dd7kz" (UID: "514e4bfb-77ef-4b1f-977a-3f9308cf886c") : secret "image-registry-tls" not found Nov 03 19:56:45.749924 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:45.749606 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:56:45.749924 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:45.749635 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert podName:ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:46.249625305 +0000 UTC m=+31.055681167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2zqfd" (UID: "ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22") : secret "networking-console-plugin-cert" not found Nov 03 19:56:45.749924 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.749896 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/433c5b6b-176b-44d1-afc0-6884045e468c-config\") pod \"console-operator-7cd56b8dbc-jk8h6\" (UID: \"433c5b6b-176b-44d1-afc0-6884045e468c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:45.750750 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:45.749978 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 19:56:45.750750 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.750450 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/433c5b6b-176b-44d1-afc0-6884045e468c-trusted-ca\") pod \"console-operator-7cd56b8dbc-jk8h6\" (UID: \"433c5b6b-176b-44d1-afc0-6884045e468c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:45.750750 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.750601 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/449a05ee-ead2-4507-8d65-faaeb3ab3281-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:45.753288 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.751570 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-config-volume\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:45.753288 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.752005 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/514e4bfb-77ef-4b1f-977a-3f9308cf886c-trusted-ca\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.753288 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.752408 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44-config\") pod \"service-ca-operator-544d749c8c-9ncj4\" (UID: \"23f64ae9-fdda-4f2e-b3a5-1c552cc55a44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" Nov 03 19:56:45.753288 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.752603 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/adc4cee9-00f0-4fac-b655-f58c21377362-registry-certificates\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.753288 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:45.752667 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs podName:653cb377-919b-4793-85a4-0d60d6aeaff6 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:46.252651098 +0000 UTC m=+31.058706958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs") pod "router-default-6bfcd89b85-9qb4s" (UID: "653cb377-919b-4793-85a4-0d60d6aeaff6") : secret "router-metrics-certs-default" not found Nov 03 19:56:45.753288 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.752888 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/adc4cee9-00f0-4fac-b655-f58c21377362-ca-trust-extracted\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.753288 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.753213 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/adc4cee9-00f0-4fac-b655-f58c21377362-installation-pull-secrets\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.753288 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.753248 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-default-certificate\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:45.754150 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.753315 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-tmp-dir\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:45.754150 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:45.753540 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:56:45.754150 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:45.753642 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls podName:fcfef4da-a4d7-4ef0-a383-ff2acb8cac79 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:46.253626732 +0000 UTC m=+31.059682603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fsrm8" (UID: "fcfef4da-a4d7-4ef0-a383-ff2acb8cac79") : secret "samples-operator-tls" not found Nov 03 19:56:45.754150 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.753931 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/449a05ee-ead2-4507-8d65-faaeb3ab3281-tmp\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:45.754498 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.754475 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/514e4bfb-77ef-4b1f-977a-3f9308cf886c-image-registry-private-configuration\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.754720 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.754690 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adc4cee9-00f0-4fac-b655-f58c21377362-trusted-ca\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.754837 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.754733 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:45.754969 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.754946 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/449a05ee-ead2-4507-8d65-faaeb3ab3281-serving-cert\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:45.755065 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:45.755000 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle podName:653cb377-919b-4793-85a4-0d60d6aeaff6 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:46.254983829 +0000 UTC m=+31.061039700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle") pod "router-default-6bfcd89b85-9qb4s" (UID: "653cb377-919b-4793-85a4-0d60d6aeaff6") : configmap references non-existent config key: service-ca.crt Nov 03 19:56:45.755127 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.755062 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/433c5b6b-176b-44d1-afc0-6884045e468c-serving-cert\") pod \"console-operator-7cd56b8dbc-jk8h6\" (UID: \"433c5b6b-176b-44d1-afc0-6884045e468c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:45.755193 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.755133 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-stats-auth\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:45.755424 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.755402 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ff051e4a-95a6-4c0d-9f69-773974d6eb7c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq\" (UID: \"ff051e4a-95a6-4c0d-9f69-773974d6eb7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" Nov 03 19:56:45.755493 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.755462 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/514e4bfb-77ef-4b1f-977a-3f9308cf886c-ca-trust-extracted\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.755564 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.755543 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/514e4bfb-77ef-4b1f-977a-3f9308cf886c-installation-pull-secrets\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.756630 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.756607 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aed0ee7d-d648-4138-b5ef-2ae6795321d2-cert\") pod \"ingress-canary-95k62\" (UID: \"aed0ee7d-d648-4138-b5ef-2ae6795321d2\") " pod="openshift-ingress-canary/ingress-canary-95k62" Nov 03 19:56:45.756884 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.756863 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44-serving-cert\") pod \"service-ca-operator-544d749c8c-9ncj4\" (UID: \"23f64ae9-fdda-4f2e-b3a5-1c552cc55a44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" Nov 03 19:56:45.756957 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.756888 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/adc4cee9-00f0-4fac-b655-f58c21377362-image-registry-private-configuration\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.767031 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.767007 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh8xj\" (UniqueName: \"kubernetes.io/projected/61f1d7a9-a8b3-4db5-878d-4b8d1632fafc-kube-api-access-jh8xj\") pod \"kas-connection-checker-685895c7c9-5dgdg\" (UID: \"61f1d7a9-a8b3-4db5-878d-4b8d1632fafc\") " pod="kube-system/kas-connection-checker-685895c7c9-5dgdg" Nov 03 19:56:45.767940 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.767916 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5sh5\" (UniqueName: \"kubernetes.io/projected/aed0ee7d-d648-4138-b5ef-2ae6795321d2-kube-api-access-w5sh5\") pod \"ingress-canary-95k62\" (UID: \"aed0ee7d-d648-4138-b5ef-2ae6795321d2\") " pod="openshift-ingress-canary/ingress-canary-95k62" Nov 03 19:56:45.768043 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.767975 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-bound-sa-token\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.769090 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.768527 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f594\" (UniqueName: \"kubernetes.io/projected/6de4fd5d-d10a-4630-80bf-3838fb02d679-kube-api-access-8f594\") pod \"network-check-source-6f78d59f99-qtbqg\" (UID: \"6de4fd5d-d10a-4630-80bf-3838fb02d679\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qtbqg" Nov 03 19:56:45.769090 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.768767 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl89h\" (UniqueName: \"kubernetes.io/projected/158d8c01-2d31-43b4-a098-0c731c1fd163-kube-api-access-wl89h\") pod \"kas-connection-checker-685895c7c9-6r7g9\" (UID: \"158d8c01-2d31-43b4-a098-0c731c1fd163\") " pod="kube-system/kas-connection-checker-685895c7c9-6r7g9" Nov 03 19:56:45.769090 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.769054 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb6j8\" (UniqueName: \"kubernetes.io/projected/653cb377-919b-4793-85a4-0d60d6aeaff6-kube-api-access-vb6j8\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:45.769575 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.769550 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd49x\" (UniqueName: \"kubernetes.io/projected/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-kube-api-access-dd49x\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:45.770119 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.770037 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtdbf\" (UniqueName: \"kubernetes.io/projected/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-kube-api-access-gtdbf\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:45.770509 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.770358 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm2tl\" (UniqueName: \"kubernetes.io/projected/ff051e4a-95a6-4c0d-9f69-773974d6eb7c-kube-api-access-gm2tl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq\" (UID: \"ff051e4a-95a6-4c0d-9f69-773974d6eb7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" Nov 03 19:56:45.770714 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.770663 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vgq9\" (UniqueName: \"kubernetes.io/projected/c60cb950-5dd9-4a2b-bba3-b3c38d3591e3-kube-api-access-6vgq9\") pod \"volume-data-source-validator-58c9f5f46d-mvxwm\" (UID: \"c60cb950-5dd9-4a2b-bba3-b3c38d3591e3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mvxwm" Nov 03 19:56:45.773398 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.771565 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml5nd\" (UniqueName: \"kubernetes.io/projected/4726eb4c-48e8-423d-a62c-250a18bd8b83-kube-api-access-ml5nd\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:45.774239 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.773615 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hb8f\" (UniqueName: \"kubernetes.io/projected/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44-kube-api-access-6hb8f\") pod \"service-ca-operator-544d749c8c-9ncj4\" (UID: \"23f64ae9-fdda-4f2e-b3a5-1c552cc55a44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" Nov 03 19:56:45.774239 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.773636 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkf49\" (UniqueName: \"kubernetes.io/projected/5a7d187d-9a17-475b-8ee4-ae5836d9a73a-kube-api-access-gkf49\") pod \"kas-connection-checker-685895c7c9-qkzrh\" (UID: \"5a7d187d-9a17-475b-8ee4-ae5836d9a73a\") " pod="kube-system/kas-connection-checker-685895c7c9-qkzrh" Nov 03 19:56:45.774239 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.773717 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-bound-sa-token\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.774239 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.773849 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djg6c\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-kube-api-access-djg6c\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:45.774986 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.774962 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sx9r\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-kube-api-access-6sx9r\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:45.775057 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.775039 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxkg4\" (UniqueName: \"kubernetes.io/projected/449a05ee-ead2-4507-8d65-faaeb3ab3281-kube-api-access-zxkg4\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:45.775783 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.775763 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x4lc\" (UniqueName: \"kubernetes.io/projected/433c5b6b-176b-44d1-afc0-6884045e468c-kube-api-access-8x4lc\") pod \"console-operator-7cd56b8dbc-jk8h6\" (UID: \"433c5b6b-176b-44d1-afc0-6884045e468c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:45.814216 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.814189 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6r7g9" Nov 03 19:56:45.823338 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.823321 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qkzrh" Nov 03 19:56:45.843568 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.843545 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:45.866785 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.866752 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:45.878821 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.878538 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5dgdg" Nov 03 19:56:45.879517 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.879450 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" Nov 03 19:56:45.903463 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.903265 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mvxwm" Nov 03 19:56:45.918558 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.918292 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" Nov 03 19:56:45.926581 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.925146 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qtbqg" Nov 03 19:56:45.930829 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:45.930806 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-95k62" Nov 03 19:56:46.118859 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.118832 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6r7g9"] Nov 03 19:56:46.121291 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.121268 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qkzrh"] Nov 03 19:56:46.121568 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:46.121522 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158d8c01_2d31_43b4_a098_0c731c1fd163.slice/crio-7149a3ee7166a852cfec10096e691606bf8c920236a0a5a1c61f6d7330c237cf WatchSource:0}: Error finding container 7149a3ee7166a852cfec10096e691606bf8c920236a0a5a1c61f6d7330c237cf: Status 404 returned error can't find the container with id 7149a3ee7166a852cfec10096e691606bf8c920236a0a5a1c61f6d7330c237cf Nov 03 19:56:46.124297 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:46.124272 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7d187d_9a17_475b_8ee4_ae5836d9a73a.slice/crio-26e2d32ed3f80f30a74bf7ab2ae1ed4231422c16e0856fa91e22aaf14e1500f8 WatchSource:0}: Error finding container 26e2d32ed3f80f30a74bf7ab2ae1ed4231422c16e0856fa91e22aaf14e1500f8: Status 404 returned error can't find the container with id 26e2d32ed3f80f30a74bf7ab2ae1ed4231422c16e0856fa91e22aaf14e1500f8 Nov 03 19:56:46.143284 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.143251 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jk8h6"] Nov 03 19:56:46.145241 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:46.145212 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433c5b6b_176b_44d1_afc0_6884045e468c.slice/crio-75a18ed8c31658475f68e433ccb3231e073366526648ddee635eba9cbe17ef84 WatchSource:0}: Error finding container 75a18ed8c31658475f68e433ccb3231e073366526648ddee635eba9cbe17ef84: Status 404 returned error can't find the container with id 75a18ed8c31658475f68e433ccb3231e073366526648ddee635eba9cbe17ef84 Nov 03 19:56:46.181406 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.180421 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zl889"] Nov 03 19:56:46.185024 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:46.184987 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449a05ee_ead2_4507_8d65_faaeb3ab3281.slice/crio-6accded0538df0201232b8f979dadc1809816fb77ad11794fc7c2dfdcf04087d WatchSource:0}: Error finding container 6accded0538df0201232b8f979dadc1809816fb77ad11794fc7c2dfdcf04087d: Status 404 returned error can't find the container with id 6accded0538df0201232b8f979dadc1809816fb77ad11794fc7c2dfdcf04087d Nov 03 19:56:46.240343 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.240318 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5dgdg"] Nov 03 19:56:46.242841 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:46.242818 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f1d7a9_a8b3_4db5_878d_4b8d1632fafc.slice/crio-80dc414f095cf57e0972d1f3e55a667bde76a7616d344e0ea893b46227440ef1 WatchSource:0}: Error finding container 80dc414f095cf57e0972d1f3e55a667bde76a7616d344e0ea893b46227440ef1: Status 404 returned error can't find the container with id 80dc414f095cf57e0972d1f3e55a667bde76a7616d344e0ea893b46227440ef1 Nov 03 19:56:46.255563 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.254163 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mvxwm"] Nov 03 19:56:46.255563 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.255370 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq"] Nov 03 19:56:46.256964 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.256942 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:46.257044 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.256987 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:46.257044 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.257022 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2zqfd\" (UID: \"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:56:46.257140 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.257088 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:46.257140 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.257120 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:46.257271 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.257167 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:46.257271 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.257208 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:46.257365 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.257268 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:46.257365 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:46.257315 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:56:46.257464 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:46.257368 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert podName:ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:47.257352457 +0000 UTC m=+32.063408323 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2zqfd" (UID: "ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22") : secret "networking-console-plugin-cert" not found Nov 03 19:56:46.257464 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:46.257424 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 19:56:46.257464 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:46.257437 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 19:56:46.257464 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:46.257461 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls podName:4726eb4c-48e8-423d-a62c-250a18bd8b83 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:47.257449064 +0000 UTC m=+32.063504927 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nb7st" (UID: "4726eb4c-48e8-423d-a62c-250a18bd8b83") : secret "cluster-monitoring-operator-tls" not found Nov 03 19:56:46.257661 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:46.257477 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs podName:653cb377-919b-4793-85a4-0d60d6aeaff6 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:47.257468744 +0000 UTC m=+32.063524604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs") pod "router-default-6bfcd89b85-9qb4s" (UID: "653cb377-919b-4793-85a4-0d60d6aeaff6") : secret "router-metrics-certs-default" not found Nov 03 19:56:46.257661 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:46.257508 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:56:46.257661 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:46.257531 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:56:46.257661 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:46.257537 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls podName:93a3cf28-8c0c-4735-b177-d28bc7c8ea3f nodeName:}" failed. No retries permitted until 2026-11-03 19:56:47.257526525 +0000 UTC m=+32.063582390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls") pod "dns-default-xsh7l" (UID: "93a3cf28-8c0c-4735-b177-d28bc7c8ea3f") : secret "dns-default-metrics-tls" not found Nov 03 19:56:46.257661 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:46.257542 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d95b6bf4b-dd7kz: secret "image-registry-tls" not found Nov 03 19:56:46.257661 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:46.257571 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls podName:514e4bfb-77ef-4b1f-977a-3f9308cf886c nodeName:}" failed. No retries permitted until 2026-11-03 19:56:47.257561456 +0000 UTC m=+32.063617319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls") pod "image-registry-5d95b6bf4b-dd7kz" (UID: "514e4bfb-77ef-4b1f-977a-3f9308cf886c") : secret "image-registry-tls" not found Nov 03 19:56:46.257661 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:46.257589 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:56:46.257661 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:46.257618 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls podName:fcfef4da-a4d7-4ef0-a383-ff2acb8cac79 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:47.257609044 +0000 UTC m=+32.063664893 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fsrm8" (UID: "fcfef4da-a4d7-4ef0-a383-ff2acb8cac79") : secret "samples-operator-tls" not found Nov 03 19:56:46.257661 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:46.257647 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle podName:653cb377-919b-4793-85a4-0d60d6aeaff6 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:47.257637428 +0000 UTC m=+32.063693292 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle") pod "router-default-6bfcd89b85-9qb4s" (UID: "653cb377-919b-4793-85a4-0d60d6aeaff6") : configmap references non-existent config key: service-ca.crt Nov 03 19:56:46.258069 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:46.257716 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:56:46.258069 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:46.257731 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9d99c5c47-ttssq: secret "image-registry-tls" not found Nov 03 19:56:46.258069 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:46.257777 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls podName:adc4cee9-00f0-4fac-b655-f58c21377362 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:47.257760873 +0000 UTC m=+32.063816735 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls") pod "image-registry-9d99c5c47-ttssq" (UID: "adc4cee9-00f0-4fac-b655-f58c21377362") : secret "image-registry-tls" not found Nov 03 19:56:46.262153 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:46.261763 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60cb950_5dd9_4a2b_bba3_b3c38d3591e3.slice/crio-fc26bdf4cad8d170f3748ba39a20387db0e062e58189b5bb3753406739d9bf26 WatchSource:0}: Error finding container fc26bdf4cad8d170f3748ba39a20387db0e062e58189b5bb3753406739d9bf26: Status 404 returned error can't find the container with id fc26bdf4cad8d170f3748ba39a20387db0e062e58189b5bb3753406739d9bf26 Nov 03 19:56:46.262991 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.262971 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-qtbqg"] Nov 03 19:56:46.275812 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.275792 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4"] Nov 03 19:56:46.288534 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.288466 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-95k62"] Nov 03 19:56:46.292540 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:46.292511 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed0ee7d_d648_4138_b5ef_2ae6795321d2.slice/crio-893a64eb71fdd26145f701505822dc8f1b865079010077d13e0ab2bc24fbfbcd WatchSource:0}: Error finding container 893a64eb71fdd26145f701505822dc8f1b865079010077d13e0ab2bc24fbfbcd: Status 404 returned error can't find the container with id 893a64eb71fdd26145f701505822dc8f1b865079010077d13e0ab2bc24fbfbcd Nov 03 19:56:46.837503 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.837470 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:46.837677 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.837565 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:46.837747 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.837679 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:46.842234 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.842197 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sztcx\"" Nov 03 19:56:46.842375 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.842258 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 19:56:46.842375 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.842340 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-72cwb\"" Nov 03 19:56:46.842375 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.842354 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hc4hl\"" Nov 03 19:56:46.842536 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:46.842437 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 19:56:47.072206 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.072123 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mvxwm" event={"ID":"c60cb950-5dd9-4a2b-bba3-b3c38d3591e3","Type":"ContainerStarted","Data":"fc26bdf4cad8d170f3748ba39a20387db0e062e58189b5bb3753406739d9bf26"} Nov 03 19:56:47.074118 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.074093 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zl889" event={"ID":"449a05ee-ead2-4507-8d65-faaeb3ab3281","Type":"ContainerStarted","Data":"6accded0538df0201232b8f979dadc1809816fb77ad11794fc7c2dfdcf04087d"} Nov 03 19:56:47.076019 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.075968 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" event={"ID":"433c5b6b-176b-44d1-afc0-6884045e468c","Type":"ContainerStarted","Data":"75a18ed8c31658475f68e433ccb3231e073366526648ddee635eba9cbe17ef84"} Nov 03 19:56:47.079053 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.079025 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qkzrh" event={"ID":"5a7d187d-9a17-475b-8ee4-ae5836d9a73a","Type":"ContainerStarted","Data":"3fa4dee365261cfea2c599bdd55ff032fa61f6829f81c8f6fdaf3b6fceb13620"} Nov 03 19:56:47.079164 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.079065 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qkzrh" event={"ID":"5a7d187d-9a17-475b-8ee4-ae5836d9a73a","Type":"ContainerStarted","Data":"26e2d32ed3f80f30a74bf7ab2ae1ed4231422c16e0856fa91e22aaf14e1500f8"} Nov 03 19:56:47.081227 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.081158 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" event={"ID":"ff051e4a-95a6-4c0d-9f69-773974d6eb7c","Type":"ContainerStarted","Data":"9d07cd3031c2ed29009a1a485c0dbd36bbb170b5754e25889aaa3d1eb9607e47"} Nov 03 19:56:47.083162 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.083082 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5dgdg" event={"ID":"61f1d7a9-a8b3-4db5-878d-4b8d1632fafc","Type":"ContainerStarted","Data":"c340ec8002870f3750d90f8b9e4a6d64b136daeed4fe093dd03a4b704b57afd8"} Nov 03 19:56:47.083162 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.083130 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5dgdg" event={"ID":"61f1d7a9-a8b3-4db5-878d-4b8d1632fafc","Type":"ContainerStarted","Data":"80dc414f095cf57e0972d1f3e55a667bde76a7616d344e0ea893b46227440ef1"} Nov 03 19:56:47.085577 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.085551 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6r7g9" event={"ID":"158d8c01-2d31-43b4-a098-0c731c1fd163","Type":"ContainerStarted","Data":"c6411a0e3db0c48e4cb034796a0ab60c967f9fcb260c2210ba070ff439ca2cf8"} Nov 03 19:56:47.085664 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.085599 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6r7g9" event={"ID":"158d8c01-2d31-43b4-a098-0c731c1fd163","Type":"ContainerStarted","Data":"7149a3ee7166a852cfec10096e691606bf8c920236a0a5a1c61f6d7330c237cf"} Nov 03 19:56:47.087447 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.087386 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" event={"ID":"23f64ae9-fdda-4f2e-b3a5-1c552cc55a44","Type":"ContainerStarted","Data":"7850e679b9b2db944c0e6df47944040d848ca279a5b6f526b886319f5a6747e1"} Nov 03 19:56:47.090588 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.090533 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-95k62" event={"ID":"aed0ee7d-d648-4138-b5ef-2ae6795321d2","Type":"ContainerStarted","Data":"893a64eb71fdd26145f701505822dc8f1b865079010077d13e0ab2bc24fbfbcd"} Nov 03 19:56:47.092345 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.092295 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qtbqg" event={"ID":"6de4fd5d-d10a-4630-80bf-3838fb02d679","Type":"ContainerStarted","Data":"da3861e8abc8d28dfb517bc237f42ec79e979910ae486a1c3c1e93c54d392277"} Nov 03 19:56:47.100800 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.099891 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-qkzrh" podStartSLOduration=418.099872892 podStartE2EDuration="6m58.099872892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:47.098803497 +0000 UTC m=+31.904859370" watchObservedRunningTime="2026-11-03 19:56:47.099872892 +0000 UTC m=+31.905928762" Nov 03 19:56:47.141276 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.140450 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5dgdg" podStartSLOduration=418.140436897 podStartE2EDuration="6m58.140436897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:47.120761791 +0000 UTC m=+31.926817664" watchObservedRunningTime="2026-11-03 19:56:47.140436897 +0000 UTC m=+31.946492816" Nov 03 19:56:47.141276 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.140628 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6r7g9" podStartSLOduration=418.140621328 podStartE2EDuration="6m58.140621328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:47.140537261 +0000 UTC m=+31.946593133" watchObservedRunningTime="2026-11-03 19:56:47.140621328 +0000 UTC m=+31.946677200" Nov 03 19:56:47.268285 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.268218 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:47.268468 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.268327 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:47.268468 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.268363 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:47.268468 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.268389 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2zqfd\" (UID: \"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:56:47.268468 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.268446 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:47.268666 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.268478 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:47.268666 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.268519 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:47.268666 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:47.268545 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:47.268800 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:47.268680 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:56:47.268800 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:47.268695 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9d99c5c47-ttssq: secret "image-registry-tls" not found Nov 03 19:56:47.268800 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:47.268754 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls podName:adc4cee9-00f0-4fac-b655-f58c21377362 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:49.268735797 +0000 UTC m=+34.074791661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls") pod "image-registry-9d99c5c47-ttssq" (UID: "adc4cee9-00f0-4fac-b655-f58c21377362") : secret "image-registry-tls" not found Nov 03 19:56:47.269197 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:47.269160 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 19:56:47.269315 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:47.269265 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs podName:653cb377-919b-4793-85a4-0d60d6aeaff6 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:49.269216729 +0000 UTC m=+34.075272581 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs") pod "router-default-6bfcd89b85-9qb4s" (UID: "653cb377-919b-4793-85a4-0d60d6aeaff6") : secret "router-metrics-certs-default" not found Nov 03 19:56:47.269416 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:47.269318 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 19:56:47.269416 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:47.269389 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls podName:4726eb4c-48e8-423d-a62c-250a18bd8b83 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:49.269340261 +0000 UTC m=+34.075396123 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nb7st" (UID: "4726eb4c-48e8-423d-a62c-250a18bd8b83") : secret "cluster-monitoring-operator-tls" not found Nov 03 19:56:47.269536 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:47.269455 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:56:47.269536 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:47.269467 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d95b6bf4b-dd7kz: secret "image-registry-tls" not found Nov 03 19:56:47.269536 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:47.269513 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls podName:514e4bfb-77ef-4b1f-977a-3f9308cf886c nodeName:}" failed. No retries permitted until 2026-11-03 19:56:49.269489447 +0000 UTC m=+34.075545319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls") pod "image-registry-5d95b6bf4b-dd7kz" (UID: "514e4bfb-77ef-4b1f-977a-3f9308cf886c") : secret "image-registry-tls" not found Nov 03 19:56:47.269690 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:47.269566 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:56:47.269690 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:47.269598 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert podName:ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:49.269588755 +0000 UTC m=+34.075644626 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2zqfd" (UID: "ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22") : secret "networking-console-plugin-cert" not found Nov 03 19:56:47.269690 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:47.269637 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:56:47.269690 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:47.269660 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls podName:93a3cf28-8c0c-4735-b177-d28bc7c8ea3f nodeName:}" failed. No retries permitted until 2026-11-03 19:56:49.269652494 +0000 UTC m=+34.075708343 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls") pod "dns-default-xsh7l" (UID: "93a3cf28-8c0c-4735-b177-d28bc7c8ea3f") : secret "dns-default-metrics-tls" not found Nov 03 19:56:47.269900 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:47.269729 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:56:47.269900 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:47.269758 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls podName:fcfef4da-a4d7-4ef0-a383-ff2acb8cac79 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:49.269748871 +0000 UTC m=+34.075804721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fsrm8" (UID: "fcfef4da-a4d7-4ef0-a383-ff2acb8cac79") : secret "samples-operator-tls" not found Nov 03 19:56:47.269900 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:47.269813 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle podName:653cb377-919b-4793-85a4-0d60d6aeaff6 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:49.269804212 +0000 UTC m=+34.075860064 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle") pod "router-default-6bfcd89b85-9qb4s" (UID: "653cb377-919b-4793-85a4-0d60d6aeaff6") : configmap references non-existent config key: service-ca.crt Nov 03 19:56:49.109430 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:49.109346 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mvxwm" event={"ID":"c60cb950-5dd9-4a2b-bba3-b3c38d3591e3","Type":"ContainerStarted","Data":"d04e0032b1245432dd5ee9d02f5bec3e6873a4d5271b4eb9c8970a371afea1ed"} Nov 03 19:56:49.297891 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:49.296402 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:49.297891 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:49.296456 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:49.297891 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:49.296486 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2zqfd\" (UID: \"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:56:49.297891 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:49.296545 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:49.297891 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:49.296578 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:49.297891 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:49.296615 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:49.297891 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:49.296641 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:49.297891 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:49.296697 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:49.297891 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:49.296818 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 19:56:49.297891 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:49.296879 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs podName:653cb377-919b-4793-85a4-0d60d6aeaff6 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:53.296860247 +0000 UTC m=+38.102916099 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs") pod "router-default-6bfcd89b85-9qb4s" (UID: "653cb377-919b-4793-85a4-0d60d6aeaff6") : secret "router-metrics-certs-default" not found Nov 03 19:56:49.297891 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:49.296947 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 19:56:49.297891 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:49.296979 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls podName:4726eb4c-48e8-423d-a62c-250a18bd8b83 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:53.296968541 +0000 UTC m=+38.103024393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nb7st" (UID: "4726eb4c-48e8-423d-a62c-250a18bd8b83") : secret "cluster-monitoring-operator-tls" not found Nov 03 19:56:49.297891 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:49.297041 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:56:49.297891 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:49.297053 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d95b6bf4b-dd7kz: secret "image-registry-tls" not found Nov 03 19:56:49.297891 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:49.297094 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls podName:514e4bfb-77ef-4b1f-977a-3f9308cf886c nodeName:}" failed. No retries permitted until 2026-11-03 19:56:53.297072909 +0000 UTC m=+38.103128759 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls") pod "image-registry-5d95b6bf4b-dd7kz" (UID: "514e4bfb-77ef-4b1f-977a-3f9308cf886c") : secret "image-registry-tls" not found Nov 03 19:56:49.297891 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:49.297151 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:56:49.298794 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:49.297201 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert podName:ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:53.29718969 +0000 UTC m=+38.103245552 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2zqfd" (UID: "ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22") : secret "networking-console-plugin-cert" not found Nov 03 19:56:49.298794 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:49.297257 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:56:49.298794 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:49.297287 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls podName:93a3cf28-8c0c-4735-b177-d28bc7c8ea3f nodeName:}" failed. No retries permitted until 2026-11-03 19:56:53.297276887 +0000 UTC m=+38.103332736 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls") pod "dns-default-xsh7l" (UID: "93a3cf28-8c0c-4735-b177-d28bc7c8ea3f") : secret "dns-default-metrics-tls" not found Nov 03 19:56:49.298794 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:49.297351 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:56:49.298794 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:49.297383 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls podName:fcfef4da-a4d7-4ef0-a383-ff2acb8cac79 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:53.297373088 +0000 UTC m=+38.103428949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fsrm8" (UID: "fcfef4da-a4d7-4ef0-a383-ff2acb8cac79") : secret "samples-operator-tls" not found Nov 03 19:56:49.298794 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:49.297449 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle podName:653cb377-919b-4793-85a4-0d60d6aeaff6 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:53.297439217 +0000 UTC m=+38.103495073 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle") pod "router-default-6bfcd89b85-9qb4s" (UID: "653cb377-919b-4793-85a4-0d60d6aeaff6") : configmap references non-existent config key: service-ca.crt Nov 03 19:56:49.298794 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:49.297508 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:56:49.298794 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:49.297517 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9d99c5c47-ttssq: secret "image-registry-tls" not found Nov 03 19:56:49.298794 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:49.297544 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls podName:adc4cee9-00f0-4fac-b655-f58c21377362 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:53.297535131 +0000 UTC m=+38.103590983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls") pod "image-registry-9d99c5c47-ttssq" (UID: "adc4cee9-00f0-4fac-b655-f58c21377362") : secret "image-registry-tls" not found Nov 03 19:56:49.397647 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:49.397566 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:49.397647 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:49.397625 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs\") pod \"network-metrics-daemon-dcqpv\" (UID: \"ab34f1da-ace1-443d-bddd-9d9905d22e52\") " pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:49.397863 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:49.397821 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 19:56:49.397930 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:49.397881 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs podName:ab34f1da-ace1-443d-bddd-9d9905d22e52 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:21.39786451 +0000 UTC m=+66.203920374 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs") pod "network-metrics-daemon-dcqpv" (UID: "ab34f1da-ace1-443d-bddd-9d9905d22e52") : secret "metrics-daemon-secret" not found Nov 03 19:56:49.402448 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:49.402419 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:49.498081 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:49.498043 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsb75\" (UniqueName: \"kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75\") pod \"network-check-target-zn2ms\" (UID: \"f90a0a4e-2794-46fc-a799-9c1593d859ab\") " pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:49.501333 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:49.501304 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsb75\" (UniqueName: \"kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75\") pod \"network-check-target-zn2ms\" (UID: \"f90a0a4e-2794-46fc-a799-9c1593d859ab\") " pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:49.563590 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:49.563551 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:49.583913 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:49.583817 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:49.897753 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:49.897677 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mvxwm" podStartSLOduration=362.485134497 podStartE2EDuration="6m4.897640641s" totalImagesPullingTime="2.412506144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:49.129505451 +0000 UTC m=+33.935561323" watchObservedRunningTime="2026-11-03 19:56:49.897640641 +0000 UTC m=+34.703696512" Nov 03 19:56:49.899484 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:49.899463 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mvsdz"] Nov 03 19:56:49.902118 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:49.902077 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e4493e_d29f_49b6_851e_cbd01214bf5c.slice/crio-e6088cca271321bf7ced864ab2bb5e69a37f875ca07627db935442ce2ebb1694 WatchSource:0}: Error finding container e6088cca271321bf7ced864ab2bb5e69a37f875ca07627db935442ce2ebb1694: Status 404 returned error can't find the container with id e6088cca271321bf7ced864ab2bb5e69a37f875ca07627db935442ce2ebb1694 Nov 03 19:56:49.926389 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:49.926320 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zn2ms"] Nov 03 19:56:49.929912 ip-10-0-139-53 kubenswrapper[2581]: W1103 19:56:49.929861 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90a0a4e_2794_46fc_a799_9c1593d859ab.slice/crio-affcf81e798d485c91871d4746b1d40e88faf886046aec06b260249516ab23cf WatchSource:0}: Error finding container affcf81e798d485c91871d4746b1d40e88faf886046aec06b260249516ab23cf: Status 404 returned error can't find the container with id affcf81e798d485c91871d4746b1d40e88faf886046aec06b260249516ab23cf Nov 03 19:56:50.114212 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:50.114115 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zn2ms" event={"ID":"f90a0a4e-2794-46fc-a799-9c1593d859ab","Type":"ContainerStarted","Data":"affcf81e798d485c91871d4746b1d40e88faf886046aec06b260249516ab23cf"} Nov 03 19:56:50.117063 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:50.116972 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mvsdz" event={"ID":"d2e4493e-d29f-49b6-851e-cbd01214bf5c","Type":"ContainerStarted","Data":"e6088cca271321bf7ced864ab2bb5e69a37f875ca07627db935442ce2ebb1694"} Nov 03 19:56:52.004356 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.004122 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k9x2f"] Nov 03 19:56:52.030209 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.027485 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-l984r-mntts"] Nov 03 19:56:52.030209 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.028229 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:52.037999 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.033678 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 19:56:52.037999 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.033895 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 19:56:52.037999 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.034028 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kfdzx\"" Nov 03 19:56:52.055375 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.055340 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k9x2f"] Nov 03 19:56:52.055493 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.055400 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-l984r-mntts"] Nov 03 19:56:52.055547 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.055490 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:52.128662 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.128574 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:52.128913 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.128897 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1fc8cec5-9248-4a47-822e-d81b37a380ef-crio-socket\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:52.129087 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.129071 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzdnd\" (UniqueName: \"kubernetes.io/projected/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-api-access-kzdnd\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:52.129300 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.129279 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:52.129423 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.129408 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1fc8cec5-9248-4a47-822e-d81b37a380ef-data-volume\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:52.129811 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.129783 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qtbqg" event={"ID":"6de4fd5d-d10a-4630-80bf-3838fb02d679","Type":"ContainerStarted","Data":"207e363c84808fad9a090be377edf8e15bd9370f8f81170255f7f81a5e910e62"} Nov 03 19:56:52.138234 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.137791 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zl889" event={"ID":"449a05ee-ead2-4507-8d65-faaeb3ab3281","Type":"ContainerStarted","Data":"f692effef71642f90293457632bf5f1c98c1b52a85d830c4979728f184683093"} Nov 03 19:56:52.140973 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.140955 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/0.log" Nov 03 19:56:52.141071 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.140987 2581 generic.go:356] "Generic (PLEG): container finished" podID="433c5b6b-176b-44d1-afc0-6884045e468c" containerID="8bc23f6b4b12b72e6827ef7d2e3d83e119fdae7dd023eb98c93687a1358e4568" exitCode=255 Nov 03 19:56:52.141071 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.141060 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" event={"ID":"433c5b6b-176b-44d1-afc0-6884045e468c","Type":"ContainerDied","Data":"8bc23f6b4b12b72e6827ef7d2e3d83e119fdae7dd023eb98c93687a1358e4568"} Nov 03 19:56:52.141469 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.141451 2581 scope.go:122] "RemoveContainer" containerID="8bc23f6b4b12b72e6827ef7d2e3d83e119fdae7dd023eb98c93687a1358e4568" Nov 03 19:56:52.143577 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.143552 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" event={"ID":"ff051e4a-95a6-4c0d-9f69-773974d6eb7c","Type":"ContainerStarted","Data":"bc41be73cadc57028af935a02e8905446331f52ffec67f21722dcc527e212f07"} Nov 03 19:56:52.147877 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.147855 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" event={"ID":"23f64ae9-fdda-4f2e-b3a5-1c552cc55a44","Type":"ContainerStarted","Data":"645219b66f36e5a9367977b6bc1310035e4ff45f0aea585089a4b49ce24a1e0d"} Nov 03 19:56:52.176264 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.176214 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qtbqg" podStartSLOduration=354.701130854 podStartE2EDuration="6m0.176192591s" totalImagesPullingTime="5.475061737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:52.153414021 +0000 UTC m=+36.959469889" watchObservedRunningTime="2026-11-03 19:56:52.176192591 +0000 UTC m=+36.982248460" Nov 03 19:56:52.222098 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.222057 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" podStartSLOduration=364.16345794 podStartE2EDuration="6m9.222043754s" totalImagesPullingTime="5.058585814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:52.194830455 +0000 UTC m=+37.000886326" watchObservedRunningTime="2026-11-03 19:56:52.222043754 +0000 UTC m=+37.028099667" Nov 03 19:56:52.222777 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.222748 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" podStartSLOduration=358.139138295 podStartE2EDuration="6m3.222738585s" totalImagesPullingTime="5.08360029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:52.221493316 +0000 UTC m=+37.027549189" watchObservedRunningTime="2026-11-03 19:56:52.222738585 +0000 UTC m=+37.028794454" Nov 03 19:56:52.230506 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.230263 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33b6188c-0202-4bc8-813f-2d189587f3aa-service-ca-bundle\") pod \"periodic-gathering-l984r-mntts\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:52.230506 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.230313 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxblp\" (UniqueName: \"kubernetes.io/projected/33b6188c-0202-4bc8-813f-2d189587f3aa-kube-api-access-gxblp\") pod \"periodic-gathering-l984r-mntts\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:52.230506 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.230328 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33b6188c-0202-4bc8-813f-2d189587f3aa-trusted-ca-bundle\") pod \"periodic-gathering-l984r-mntts\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:52.230506 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.230413 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:52.230506 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.230463 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1fc8cec5-9248-4a47-822e-d81b37a380ef-crio-socket\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:52.230812 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.230538 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzdnd\" (UniqueName: \"kubernetes.io/projected/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-api-access-kzdnd\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:52.230812 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.230576 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/33b6188c-0202-4bc8-813f-2d189587f3aa-archives-path\") pod \"periodic-gathering-l984r-mntts\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:52.230812 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.230610 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:52.230812 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.230642 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1fc8cec5-9248-4a47-822e-d81b37a380ef-data-volume\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:52.230985 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.230886 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1fc8cec5-9248-4a47-822e-d81b37a380ef-data-volume\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:52.231857 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.231794 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1fc8cec5-9248-4a47-822e-d81b37a380ef-crio-socket\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:52.233305 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:52.233278 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 19:56:52.233390 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:52.233348 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm podName:1fc8cec5-9248-4a47-822e-d81b37a380ef nodeName:}" failed. No retries permitted until 2026-11-03 19:56:52.733330818 +0000 UTC m=+37.539386683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k9x2f" (UID: "1fc8cec5-9248-4a47-822e-d81b37a380ef") : configmap "kube-rbac-proxy" not found Nov 03 19:56:52.233810 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:52.233787 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:56:52.233897 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:52.233844 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls podName:1fc8cec5-9248-4a47-822e-d81b37a380ef nodeName:}" failed. No retries permitted until 2026-11-03 19:56:52.733828268 +0000 UTC m=+37.539884120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k9x2f" (UID: "1fc8cec5-9248-4a47-822e-d81b37a380ef") : secret "insights-runtime-extractor-tls" not found Nov 03 19:56:52.257136 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.257071 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzdnd\" (UniqueName: \"kubernetes.io/projected/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-api-access-kzdnd\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:52.331791 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.331761 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxblp\" (UniqueName: \"kubernetes.io/projected/33b6188c-0202-4bc8-813f-2d189587f3aa-kube-api-access-gxblp\") pod \"periodic-gathering-l984r-mntts\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:52.331791 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.331797 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33b6188c-0202-4bc8-813f-2d189587f3aa-trusted-ca-bundle\") pod \"periodic-gathering-l984r-mntts\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:52.331980 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.331968 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/33b6188c-0202-4bc8-813f-2d189587f3aa-archives-path\") pod \"periodic-gathering-l984r-mntts\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:52.332034 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.332024 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33b6188c-0202-4bc8-813f-2d189587f3aa-service-ca-bundle\") pod \"periodic-gathering-l984r-mntts\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:52.332697 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.332672 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33b6188c-0202-4bc8-813f-2d189587f3aa-service-ca-bundle\") pod \"periodic-gathering-l984r-mntts\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:52.334369 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.333902 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33b6188c-0202-4bc8-813f-2d189587f3aa-trusted-ca-bundle\") pod \"periodic-gathering-l984r-mntts\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:52.335052 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.335030 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/33b6188c-0202-4bc8-813f-2d189587f3aa-archives-path\") pod \"periodic-gathering-l984r-mntts\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:52.359831 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.359775 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxblp\" (UniqueName: \"kubernetes.io/projected/33b6188c-0202-4bc8-813f-2d189587f3aa-kube-api-access-gxblp\") pod \"periodic-gathering-l984r-mntts\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:52.419314 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.419278 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:52.741635 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.739449 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-zl889" podStartSLOduration=355.589300248 podStartE2EDuration="6m0.73943481s" totalImagesPullingTime="5.150134562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:52.248100524 +0000 UTC m=+37.054156396" watchObservedRunningTime="2026-11-03 19:56:52.73943481 +0000 UTC m=+37.545490684" Nov 03 19:56:52.741635 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.741208 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:52.741635 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.741310 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-l984r-mntts"] Nov 03 19:56:52.741635 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:52.741349 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:52.741635 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:52.741470 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 19:56:52.741635 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:52.741529 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm podName:1fc8cec5-9248-4a47-822e-d81b37a380ef nodeName:}" failed. No retries permitted until 2026-11-03 19:56:53.741513198 +0000 UTC m=+38.547569062 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k9x2f" (UID: "1fc8cec5-9248-4a47-822e-d81b37a380ef") : configmap "kube-rbac-proxy" not found Nov 03 19:56:52.741635 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:52.741546 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:56:52.741635 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:52.741605 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls podName:1fc8cec5-9248-4a47-822e-d81b37a380ef nodeName:}" failed. No retries permitted until 2026-11-03 19:56:53.741591698 +0000 UTC m=+38.547647559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k9x2f" (UID: "1fc8cec5-9248-4a47-822e-d81b37a380ef") : secret "insights-runtime-extractor-tls" not found Nov 03 19:56:53.154009 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:53.153977 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/1.log" Nov 03 19:56:53.154873 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:53.154850 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/0.log" Nov 03 19:56:53.154993 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:53.154896 2581 generic.go:356] "Generic (PLEG): container finished" podID="433c5b6b-176b-44d1-afc0-6884045e468c" containerID="33c82892c5f4b399f0d1f3ee43a5aee3b52b13b00868cd4bbdb8ac251bc91a2f" exitCode=255 Nov 03 19:56:53.154993 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:53.154979 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" event={"ID":"433c5b6b-176b-44d1-afc0-6884045e468c","Type":"ContainerDied","Data":"33c82892c5f4b399f0d1f3ee43a5aee3b52b13b00868cd4bbdb8ac251bc91a2f"} Nov 03 19:56:53.155088 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:53.155039 2581 scope.go:122] "RemoveContainer" containerID="8bc23f6b4b12b72e6827ef7d2e3d83e119fdae7dd023eb98c93687a1358e4568" Nov 03 19:56:53.155256 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:53.155238 2581 scope.go:122] "RemoveContainer" containerID="33c82892c5f4b399f0d1f3ee43a5aee3b52b13b00868cd4bbdb8ac251bc91a2f" Nov 03 19:56:53.155463 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.155444 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jk8h6_openshift-console-operator(433c5b6b-176b-44d1-afc0-6884045e468c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" podUID="433c5b6b-176b-44d1-afc0-6884045e468c" Nov 03 19:56:53.157473 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:53.157082 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-l984r-mntts" event={"ID":"33b6188c-0202-4bc8-813f-2d189587f3aa","Type":"ContainerStarted","Data":"4dea61c608711b32be419f6e4b8734e863f0ca5346d7ad77075c0833a1d33de8"} Nov 03 19:56:53.157473 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:53.157124 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-l984r-mntts" event={"ID":"33b6188c-0202-4bc8-813f-2d189587f3aa","Type":"ContainerStarted","Data":"1ca9416f4f2401864659a5a0f7debfb2d9b3f44c8f9b1ff02a8fa40dca952bb5"} Nov 03 19:56:53.197822 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:53.197776 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-l984r-mntts" podStartSLOduration=2.197762319 podStartE2EDuration="2.197762319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:53.197072467 +0000 UTC m=+38.003128339" watchObservedRunningTime="2026-11-03 19:56:53.197762319 +0000 UTC m=+38.003818190" Nov 03 19:56:53.350224 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:53.347251 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:53.350224 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:53.347313 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:53.350224 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:53.347404 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:53.350224 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:53.347641 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:53.350224 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:53.347674 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:53.350224 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:53.347701 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2zqfd\" (UID: \"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:56:53.350224 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:53.347830 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:53.350224 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:53.347866 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:53.350224 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.347993 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:56:53.350224 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.348046 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls podName:fcfef4da-a4d7-4ef0-a383-ff2acb8cac79 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:01.3480288 +0000 UTC m=+46.154084652 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fsrm8" (UID: "fcfef4da-a4d7-4ef0-a383-ff2acb8cac79") : secret "samples-operator-tls" not found Nov 03 19:56:53.350224 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.348115 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle podName:653cb377-919b-4793-85a4-0d60d6aeaff6 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:01.348104491 +0000 UTC m=+46.154160356 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle") pod "router-default-6bfcd89b85-9qb4s" (UID: "653cb377-919b-4793-85a4-0d60d6aeaff6") : configmap references non-existent config key: service-ca.crt Nov 03 19:56:53.350224 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.348479 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:56:53.350224 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.348493 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9d99c5c47-ttssq: secret "image-registry-tls" not found Nov 03 19:56:53.350224 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.348529 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls podName:adc4cee9-00f0-4fac-b655-f58c21377362 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:01.348515982 +0000 UTC m=+46.154571845 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls") pod "image-registry-9d99c5c47-ttssq" (UID: "adc4cee9-00f0-4fac-b655-f58c21377362") : secret "image-registry-tls" not found Nov 03 19:56:53.350224 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.349541 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 19:56:53.350937 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.349580 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs podName:653cb377-919b-4793-85a4-0d60d6aeaff6 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:01.349569175 +0000 UTC m=+46.155625025 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs") pod "router-default-6bfcd89b85-9qb4s" (UID: "653cb377-919b-4793-85a4-0d60d6aeaff6") : secret "router-metrics-certs-default" not found Nov 03 19:56:53.350937 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.349801 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 19:56:53.350937 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.349849 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls podName:4726eb4c-48e8-423d-a62c-250a18bd8b83 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:01.349833483 +0000 UTC m=+46.155889346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nb7st" (UID: "4726eb4c-48e8-423d-a62c-250a18bd8b83") : secret "cluster-monitoring-operator-tls" not found Nov 03 19:56:53.350937 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.349915 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:56:53.350937 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.349925 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d95b6bf4b-dd7kz: secret "image-registry-tls" not found Nov 03 19:56:53.350937 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.349957 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls podName:514e4bfb-77ef-4b1f-977a-3f9308cf886c nodeName:}" failed. No retries permitted until 2026-11-03 19:57:01.349946698 +0000 UTC m=+46.156002570 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls") pod "image-registry-5d95b6bf4b-dd7kz" (UID: "514e4bfb-77ef-4b1f-977a-3f9308cf886c") : secret "image-registry-tls" not found Nov 03 19:56:53.350937 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.350011 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:56:53.350937 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.350044 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert podName:ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:01.350032375 +0000 UTC m=+46.156088250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2zqfd" (UID: "ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22") : secret "networking-console-plugin-cert" not found Nov 03 19:56:53.352393 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.352373 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:56:53.352500 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.352426 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls podName:93a3cf28-8c0c-4735-b177-d28bc7c8ea3f nodeName:}" failed. No retries permitted until 2026-11-03 19:57:01.352412283 +0000 UTC m=+46.158468131 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls") pod "dns-default-xsh7l" (UID: "93a3cf28-8c0c-4735-b177-d28bc7c8ea3f") : secret "dns-default-metrics-tls" not found Nov 03 19:56:53.751289 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:53.751255 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:53.751466 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:53.751358 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:53.751466 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.751374 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 19:56:53.751466 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.751458 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm podName:1fc8cec5-9248-4a47-822e-d81b37a380ef nodeName:}" failed. No retries permitted until 2026-11-03 19:56:55.751424593 +0000 UTC m=+40.557480464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k9x2f" (UID: "1fc8cec5-9248-4a47-822e-d81b37a380ef") : configmap "kube-rbac-proxy" not found Nov 03 19:56:53.751578 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.751468 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:56:53.751578 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:53.751507 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls podName:1fc8cec5-9248-4a47-822e-d81b37a380ef nodeName:}" failed. No retries permitted until 2026-11-03 19:56:55.751496617 +0000 UTC m=+40.557552464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k9x2f" (UID: "1fc8cec5-9248-4a47-822e-d81b37a380ef") : secret "insights-runtime-extractor-tls" not found Nov 03 19:56:54.162975 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:54.162947 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/1.log" Nov 03 19:56:54.163480 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:54.163368 2581 scope.go:122] "RemoveContainer" containerID="33c82892c5f4b399f0d1f3ee43a5aee3b52b13b00868cd4bbdb8ac251bc91a2f" Nov 03 19:56:54.163614 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:54.163585 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jk8h6_openshift-console-operator(433c5b6b-176b-44d1-afc0-6884045e468c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" podUID="433c5b6b-176b-44d1-afc0-6884045e468c" Nov 03 19:56:54.164833 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:54.164805 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zn2ms" event={"ID":"f90a0a4e-2794-46fc-a799-9c1593d859ab","Type":"ContainerStarted","Data":"91e166c77f65dc79f71ae3d02ea74bb26f8ebd6df5266d735905614480294a75"} Nov 03 19:56:54.164963 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:54.164863 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:54.166399 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:54.166370 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-95k62" event={"ID":"aed0ee7d-d648-4138-b5ef-2ae6795321d2","Type":"ContainerStarted","Data":"9a44c986728c60458d76ac3419d25973da5e8a8e8ed1f4300bbc867dafd6afbf"} Nov 03 19:56:54.205517 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:54.205466 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-95k62" podStartSLOduration=2.242592804 podStartE2EDuration="9.205453376s" totalImagesPullingTime="6.962860572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:54.20473906 +0000 UTC m=+39.010794930" watchObservedRunningTime="2026-11-03 19:56:54.205453376 +0000 UTC m=+39.011509245" Nov 03 19:56:54.224615 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:54.224564 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zn2ms" podStartSLOduration=34.381148876 podStartE2EDuration="38.224547206s" totalImagesPullingTime="3.84339833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:54.224335049 +0000 UTC m=+39.030390919" watchObservedRunningTime="2026-11-03 19:56:54.224547206 +0000 UTC m=+39.030603078" Nov 03 19:56:54.509257 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:54.509166 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ksm7q_81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40/dns-node-resolver/0.log" Nov 03 19:56:55.771222 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:55.771158 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:55.771634 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:55.771272 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 19:56:55.771634 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:55.771340 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:55.771634 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:55.771358 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm podName:1fc8cec5-9248-4a47-822e-d81b37a380ef nodeName:}" failed. No retries permitted until 2026-11-03 19:56:59.771335577 +0000 UTC m=+44.577391434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k9x2f" (UID: "1fc8cec5-9248-4a47-822e-d81b37a380ef") : configmap "kube-rbac-proxy" not found Nov 03 19:56:55.771634 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:55.771440 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:56:55.771634 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:55.771492 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls podName:1fc8cec5-9248-4a47-822e-d81b37a380ef nodeName:}" failed. No retries permitted until 2026-11-03 19:56:59.771475835 +0000 UTC m=+44.577531696 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k9x2f" (UID: "1fc8cec5-9248-4a47-822e-d81b37a380ef") : secret "insights-runtime-extractor-tls" not found Nov 03 19:56:55.844408 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:55.844369 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:55.844579 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:55.844417 2581 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:55.844770 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:55.844754 2581 scope.go:122] "RemoveContainer" containerID="33c82892c5f4b399f0d1f3ee43a5aee3b52b13b00868cd4bbdb8ac251bc91a2f" Nov 03 19:56:55.844997 ip-10-0-139-53 kubenswrapper[2581]: E1103 19:56:55.844975 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jk8h6_openshift-console-operator(433c5b6b-176b-44d1-afc0-6884045e468c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" podUID="433c5b6b-176b-44d1-afc0-6884045e468c" Nov 03 19:56:55.909257 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:55.909230 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jjkqt_26ac4e9f-3445-4994-aa1b-086b39b8c1cc/node-ca/0.log" Nov 03 19:56:56.309299 ip-10-0-139-53 kubenswrapper[2581]: I1103 19:56:56.309263 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-95k62_aed0ee7d-d648-4138-b5ef-2ae6795321d2/serve-healthcheck-canary/0.log" Nov 03 19:56:56.999846 ip-10-0-139-53 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 19:56:57.022658 ip-10-0-139-53 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 19:56:57.022842 ip-10-0-139-53 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 19:56:57.023018 ip-10-0-139-53 systemd[1]: kubelet.service: Consumed 3.041s CPU time. Nov 03 19:56:57.037651 ip-10-0-139-53 systemd[1]: Starting Kubernetes Kubelet... Nov 03 19:56:57.128339 ip-10-0-139-53 kubenswrapper[4918]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:56:57.128339 ip-10-0-139-53 kubenswrapper[4918]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 19:56:57.128339 ip-10-0-139-53 kubenswrapper[4918]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:56:57.128339 ip-10-0-139-53 kubenswrapper[4918]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:56:57.131280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131187 4918 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:57.131280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131274 4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:57.131280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131279 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:57.131280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131282 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:57.131280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131285 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:57.131280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131289 4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:57.131617 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131292 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:57.131617 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131295 4918 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:57.131617 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131298 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:57.131617 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131300 4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:57.131617 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131303 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:57.131617 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131306 4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:57.131617 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131309 4918 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:57.131617 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131312 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:57.131617 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131314 4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:57.131617 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131317 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:57.131617 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131319 4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:57.131617 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131322 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:57.131617 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131333 4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:57.131617 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131336 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:57.131617 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131338 4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:57.131617 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131341 4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:57.131617 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131343 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:57.131617 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131346 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:57.131617 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131349 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:57.131617 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131352 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:57.132162 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131354 4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:57.132162 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131357 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:57.132162 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131360 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:57.132162 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131363 4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:57.132162 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131365 4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:57.132162 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131368 4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:57.132162 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131370 4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:57.132162 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131373 4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:57.132162 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131376 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:57.132162 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131378 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:57.132162 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131381 4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:57.132162 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131384 4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:57.132162 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131387 4918 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:57.132162 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131390 4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:57.132162 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131393 4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:57.132162 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131395 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:57.132162 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131398 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:57.132162 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131401 4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:57.132162 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131403 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:57.132671 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131406 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:57.132671 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131409 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:57.132671 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131411 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:57.132671 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131414 4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:57.132671 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131418 4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:57.132671 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131423 4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:57.132671 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131426 4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:57.132671 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131429 4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:57.132671 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131432 4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:57.132671 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131435 4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:57.132671 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131438 4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:57.132671 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131440 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:57.132671 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131443 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:57.132671 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131446 4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:57.132671 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131448 4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:57.132671 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131451 4918 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:57.132671 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131453 4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:57.132671 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131456 4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:57.132671 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131459 4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:57.133147 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131461 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:57.133147 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131464 4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:57.133147 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131468 4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:57.133147 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131472 4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:57.133147 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131476 4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:57.133147 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131479 4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:57.133147 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131482 4918 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:57.133147 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131484 4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:57.133147 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131487 4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:57.133147 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131489 4918 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:57.133147 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131492 4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:57.133147 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131494 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:57.133147 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131497 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:57.133147 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131500 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:57.133147 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131502 4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:57.133147 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131505 4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:57.133147 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131507 4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:57.133147 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131510 4918 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:57.133147 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131513 4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:57.133699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131519 4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:57.133699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131522 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:57.133699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131524 4918 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:57.133699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131527 4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:57.133699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131530 4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:57.133699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131532 4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:57.133699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131535 4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:57.133699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131537 4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:57.133699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131540 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:57.133699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131543 4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:57.133699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131545 4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:57.133699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131548 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:57.133699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131551 4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:57.133699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131553 4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:57.133699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131556 4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:57.133699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131558 4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:57.133699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131561 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:57.133699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131564 4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:57.133699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131567 4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:57.134204 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131571 4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:57.134204 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131573 4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:57.134204 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131576 4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:57.134204 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.131579 4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:57.134204 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132358 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:57.134204 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132365 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:57.134204 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132368 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:57.134204 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132371 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:57.134204 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132374 4918 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:57.134204 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132377 4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:57.134204 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132379 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:57.134204 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132382 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:57.134204 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132385 4918 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:57.134204 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132388 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:57.134204 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132391 4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:57.134204 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132394 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:57.134204 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132397 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:57.134204 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132399 4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:57.134204 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132402 4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:57.134699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132404 4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:57.134699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132407 4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:57.134699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132410 4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:57.134699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132412 4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:57.134699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132415 4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:57.134699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132417 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:57.134699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132420 4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:57.134699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132423 4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:57.134699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132425 4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:57.134699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132428 4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:57.134699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132431 4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:57.134699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132434 4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:57.134699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132437 4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:57.134699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132440 4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:57.134699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132443 4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:57.134699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132446 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:57.134699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132448 4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:57.134699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132451 4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:57.134699 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132454 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:57.135199 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132456 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:57.135199 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132459 4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:57.135199 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132462 4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:57.135199 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132464 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:57.135199 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132467 4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:57.135199 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132469 4918 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:57.135199 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132472 4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:57.135199 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132474 4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:57.135199 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132477 4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:57.135199 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132480 4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:57.135199 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132483 4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:57.135199 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132485 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:57.135199 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132488 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:57.135199 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132491 4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:57.135199 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132494 4918 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:57.135199 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132496 4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:57.135199 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132500 4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:57.135199 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132505 4918 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:57.135199 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132507 4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:57.135726 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132521 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:57.135726 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132524 4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:57.135726 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132526 4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:57.135726 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132529 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:57.135726 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132532 4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:57.135726 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132534 4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:57.135726 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132537 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:57.135726 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132540 4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:57.135726 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132543 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:57.135726 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132546 4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:57.135726 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132549 4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:57.135726 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132551 4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:57.135726 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132554 4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:57.135726 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132557 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:57.135726 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132560 4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:57.135726 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132563 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:57.135726 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132565 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:57.135726 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132568 4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:57.135726 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132571 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:57.136226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132573 4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:57.136226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132576 4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:57.136226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132578 4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:57.136226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132581 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:57.136226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132584 4918 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:57.136226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132587 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:57.136226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132590 4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:57.136226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132593 4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:57.136226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132595 4918 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:57.136226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132598 4918 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:57.136226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132601 4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:57.136226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132603 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:57.136226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132606 4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:57.136226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132608 4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:57.136226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132611 4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:57.136226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132615 4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:57.136226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132618 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:57.136226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132621 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:57.136226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132623 4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:57.136226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132626 4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:57.136724 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132629 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:57.136724 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132631 4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:57.136724 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132634 4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:57.136724 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132637 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:57.136724 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132639 4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:57.136724 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132642 4918 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:57.136724 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132644 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:57.136724 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132647 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:57.136724 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132649 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:57.136724 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132655 4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:57.136724 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132657 4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:57.136724 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132660 4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:57.136724 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132662 4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:57.136724 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.132665 4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:57.136724 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133148 4918 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:57.136724 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133153 4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:57.136724 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133160 4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:57.136724 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133164 4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:57.136724 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133167 4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:57.137226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133169 4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:57.137226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133172 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:57.137226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133189 4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:57.137226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133192 4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:57.137226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133195 4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:57.137226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133198 4918 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:57.137226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133201 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:57.137226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133204 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:57.137226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133207 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:57.137226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133209 4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:57.137226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133212 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:57.137226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133215 4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:57.137226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133218 4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:57.137226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133221 4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:57.137226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133224 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:57.137226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133226 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:57.137226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133229 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:57.137226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133232 4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:57.137226 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133234 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:57.137719 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133237 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:57.137719 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133239 4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:57.137719 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133242 4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:57.137719 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133246 4918 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:57.137719 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133249 4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:57.137719 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133252 4918 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:57.137719 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133255 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:57.137719 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133258 4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:57.137719 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133261 4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:57.137719 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133263 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:57.137719 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133266 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:57.137719 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133269 4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:57.137719 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133272 4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:57.137719 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133274 4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:57.137719 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133277 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:57.137719 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133279 4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:57.137719 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133282 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:57.137719 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133284 4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:57.137719 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133287 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:57.137719 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133289 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:57.138253 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133292 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:57.138253 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133294 4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:57.138253 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133297 4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:57.138253 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133299 4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:57.138253 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133302 4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:57.138253 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133304 4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:57.138253 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133307 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:57.138253 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133310 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:57.138253 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133313 4918 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:57.138253 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133316 4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:57.138253 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133318 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:57.138253 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133321 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:57.138253 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133323 4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:57.138253 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133326 4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:57.138253 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133329 4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:57.138253 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133333 4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:57.138253 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133335 4918 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:57.138253 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133338 4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:57.138253 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133340 4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:57.138253 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133343 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:57.138927 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133346 4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:57.138927 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133348 4918 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:57.138927 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133351 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:57.138927 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133354 4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:57.138927 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133357 4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:57.138927 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133360 4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:57.138927 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133362 4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:57.138927 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133365 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:57.138927 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133369 4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:57.138927 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133372 4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:57.138927 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133375 4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:57.138927 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133378 4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:57.138927 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133380 4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:57.138927 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133383 4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:57.138927 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133385 4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:57.138927 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133388 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:57.138927 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133390 4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:57.138927 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133393 4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:57.138927 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133396 4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:57.139421 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133398 4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:57.139421 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133401 4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:57.139421 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133404 4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:57.139421 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133406 4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:57.139421 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133408 4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:57.139421 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133411 4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:57.139421 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133413 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:57.139421 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133416 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:57.139421 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133420 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:57.139421 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133423 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:57.139421 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133425 4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:57.139421 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133428 4918 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:57.139421 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133431 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:57.139421 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133433 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:57.139421 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133436 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:57.139421 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133439 4918 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:57.139421 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133441 4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:57.139421 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133444 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:57.139421 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133449 4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133453 4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133456 4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133459 4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.133461 4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133555 4918 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133562 4918 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133569 4918 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133574 4918 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133578 4918 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133581 4918 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133585 4918 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133589 4918 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133593 4918 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133596 4918 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133599 4918 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133603 4918 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133606 4918 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133609 4918 flags.go:64] FLAG: --cgroup-root="" Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133612 4918 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133615 4918 flags.go:64] FLAG: --client-ca-file="" Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133618 4918 flags.go:64] FLAG: --cloud-provider="external" Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133622 4918 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 19:56:57.139920 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133626 4918 flags.go:64] FLAG: --cluster-domain="" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133629 4918 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133632 4918 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133635 4918 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133638 4918 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133642 4918 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133645 4918 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133649 4918 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133652 4918 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133655 4918 flags.go:64] FLAG: --contention-profiling="false" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133659 4918 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133663 4918 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133666 4918 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133669 4918 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133673 4918 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133676 4918 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133679 4918 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133682 4918 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133685 4918 flags.go:64] FLAG: --enable-server="true" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133688 4918 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133693 4918 flags.go:64] FLAG: --event-burst="100" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133698 4918 flags.go:64] FLAG: --event-qps="50" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133701 4918 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133704 4918 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 19:56:57.140512 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133708 4918 flags.go:64] FLAG: --eviction-hard="" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133712 4918 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133715 4918 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133718 4918 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133722 4918 flags.go:64] FLAG: --eviction-soft="" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133725 4918 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133727 4918 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133732 4918 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133735 4918 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133738 4918 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133741 4918 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133744 4918 flags.go:64] FLAG: --feature-gates="" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133748 4918 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133751 4918 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133754 4918 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133758 4918 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133761 4918 flags.go:64] FLAG: --healthz-port="10248" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133764 4918 flags.go:64] FLAG: --help="false" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133767 4918 flags.go:64] FLAG: --hostname-override="ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133771 4918 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133774 4918 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133777 4918 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133780 4918 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 19:56:57.141131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133784 4918 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133787 4918 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133790 4918 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133792 4918 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133795 4918 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133798 4918 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133802 4918 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133805 4918 flags.go:64] FLAG: --kube-reserved="" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133808 4918 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133811 4918 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133814 4918 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133817 4918 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133820 4918 flags.go:64] FLAG: --lock-file="" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133823 4918 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133826 4918 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133829 4918 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133835 4918 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133838 4918 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133842 4918 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133844 4918 flags.go:64] FLAG: --logging-format="text" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133847 4918 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133851 4918 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133854 4918 flags.go:64] FLAG: --manifest-url="" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133857 4918 flags.go:64] FLAG: --manifest-url-header="" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133862 4918 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 19:56:57.141709 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133865 4918 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133868 4918 flags.go:64] FLAG: --max-pods="110" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133872 4918 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133875 4918 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133878 4918 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133881 4918 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133884 4918 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133887 4918 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133890 4918 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133897 4918 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133900 4918 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133903 4918 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133906 4918 flags.go:64] FLAG: --pod-cidr="" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133910 4918 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133913 4918 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133916 4918 flags.go:64] FLAG: --pods-per-core="0" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133919 4918 flags.go:64] FLAG: --port="10250" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133923 4918 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133926 4918 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0a62655d7020a54f6" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133929 4918 flags.go:64] FLAG: --qos-reserved="" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133932 4918 flags.go:64] FLAG: --read-only-port="10255" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133935 4918 flags.go:64] FLAG: --register-node="true" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133938 4918 flags.go:64] FLAG: --register-with-taints="" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133943 4918 flags.go:64] FLAG: --registry-burst="10" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133946 4918 flags.go:64] FLAG: --registry-qps="5" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133948 4918 flags.go:64] FLAG: --reserved-cpus="" Nov 03 19:56:57.142361 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133951 4918 flags.go:64] FLAG: --reserved-memory="" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133955 4918 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133958 4918 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133961 4918 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133964 4918 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133967 4918 flags.go:64] FLAG: --runonce="false" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133970 4918 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133974 4918 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133977 4918 flags.go:64] FLAG: --seccomp-default="false" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133980 4918 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133983 4918 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133987 4918 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133990 4918 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133993 4918 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133996 4918 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.133999 4918 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.134002 4918 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.134005 4918 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.134008 4918 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.134012 4918 flags.go:64] FLAG: --system-cgroups="" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.134015 4918 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.134020 4918 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.134023 4918 flags.go:64] FLAG: --tls-cert-file="" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.134026 4918 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.134030 4918 flags.go:64] FLAG: --tls-min-version="" Nov 03 19:56:57.143042 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.134033 4918 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.134036 4918 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.134038 4918 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.134042 4918 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.134046 4918 flags.go:64] FLAG: --v="2" Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.134050 4918 flags.go:64] FLAG: --version="false" Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.134054 4918 flags.go:64] FLAG: --vmodule="" Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.134058 4918 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.134061 4918 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134214 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134219 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134223 4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134225 4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134228 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134231 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134234 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134236 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134239 4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134242 4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134245 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134248 4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134251 4918 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:57.143941 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134253 4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:57.144810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134256 4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:57.144810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134258 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:57.144810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134261 4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:57.144810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134265 4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:57.144810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134267 4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:57.144810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134270 4918 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:57.144810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134273 4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:57.144810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134275 4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:57.144810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134278 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:57.144810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134281 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:57.144810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134284 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:57.144810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134286 4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:57.144810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134289 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:57.144810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134293 4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:57.144810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134296 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:57.144810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134298 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:57.144810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134301 4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:57.144810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134304 4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:57.144810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134306 4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:57.144810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134309 4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:57.145551 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134312 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:57.145551 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134314 4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:57.145551 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134317 4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:57.145551 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134320 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:57.145551 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134322 4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:57.145551 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134325 4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:57.145551 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134327 4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:57.145551 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134330 4918 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:57.145551 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134334 4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:57.145551 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134338 4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:57.145551 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134341 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:57.145551 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134345 4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:57.145551 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134348 4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:57.145551 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134350 4918 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:57.145551 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134353 4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:57.145551 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134357 4918 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:57.145551 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134360 4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:57.145551 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134363 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:57.145551 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134365 4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:57.146192 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134368 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:57.146192 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134371 4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:57.146192 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134373 4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:57.146192 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134376 4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:57.146192 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134379 4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:57.146192 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134382 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:57.146192 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134386 4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:57.146192 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134388 4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:57.146192 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134391 4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:57.146192 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134394 4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:57.146192 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134397 4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:57.146192 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134399 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:57.146192 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134402 4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:57.146192 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134404 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:57.146192 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134407 4918 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:57.146192 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134410 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:57.146192 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134412 4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:57.146192 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134415 4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:57.146192 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134418 4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:57.146192 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134420 4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:57.146764 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134423 4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:57.146764 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134425 4918 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:57.146764 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134428 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:57.146764 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134431 4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:57.146764 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134434 4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:57.146764 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134436 4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:57.146764 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134439 4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:57.146764 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134442 4918 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:57.146764 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134445 4918 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:57.146764 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134448 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:57.146764 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134451 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:57.146764 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134453 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:57.146764 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134456 4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:57.146764 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134458 4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:57.146764 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134461 4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:57.146764 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134464 4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:57.146764 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134466 4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:57.146764 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134468 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:57.146764 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134472 4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:57.146764 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134475 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:57.147331 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134478 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:57.147331 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134481 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:57.147331 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134483 4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:57.147331 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134487 4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:57.147331 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134491 4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:57.147331 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134495 4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:57.147331 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134498 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:57.147331 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134502 4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:57.147331 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134504 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:57.147331 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134507 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:57.147331 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134510 4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:57.147331 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134512 4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:57.147331 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134515 4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:57.147331 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.134541 4918 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:56:57.147331 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134583 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:57.147790 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134588 4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:57.147790 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134591 4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:57.147790 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134594 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:57.147790 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134597 4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:57.147790 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134599 4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:57.147790 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134604 4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:57.147790 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134607 4918 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:57.147790 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134611 4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:57.147790 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134614 4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:57.147790 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134618 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:57.147790 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134620 4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:57.147790 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134623 4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:57.147790 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134626 4918 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:57.147790 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134629 4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:57.147790 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134631 4918 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:57.147790 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134634 4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:57.147790 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134638 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:57.147790 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134641 4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:57.147790 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134644 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:57.148447 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134646 4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:57.148447 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134649 4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:57.148447 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134652 4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:57.148447 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134654 4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:57.148447 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134657 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:57.148447 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134659 4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:57.148447 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134662 4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:57.148447 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134664 4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:57.148447 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134667 4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:57.148447 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134670 4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:57.148447 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134672 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:57.148447 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134675 4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:57.148447 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134677 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:57.148447 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134680 4918 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:57.148447 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134683 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:57.148447 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134685 4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:57.148447 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134688 4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:57.148447 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134698 4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:57.148447 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134705 4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:57.148447 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134707 4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:57.148954 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134710 4918 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:57.148954 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134713 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:57.148954 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134716 4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:57.148954 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134719 4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:57.148954 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134721 4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:57.148954 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134724 4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:57.148954 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134727 4918 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:57.148954 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134729 4918 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:57.148954 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134732 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:57.148954 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134736 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:57.148954 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134738 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:57.148954 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134741 4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:57.148954 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134744 4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:57.148954 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134746 4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:57.148954 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134749 4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:57.148954 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134752 4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:57.148954 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134754 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:57.148954 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134757 4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:57.148954 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134759 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:57.148954 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134762 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:57.149467 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134764 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:57.149467 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134767 4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:57.149467 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134772 4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:57.149467 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134775 4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:57.149467 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134778 4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:57.149467 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134782 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:57.149467 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134784 4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:57.149467 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134787 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:57.149467 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134790 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:57.149467 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134793 4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:57.149467 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134797 4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:57.149467 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134799 4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:57.149467 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134802 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:57.149467 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134804 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:57.149467 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134807 4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:57.149467 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134810 4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:57.149467 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134812 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:57.149467 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134815 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:57.149467 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134817 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:57.149924 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134820 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:57.149924 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134822 4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:57.149924 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134825 4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:57.149924 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134827 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:57.149924 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134830 4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:57.149924 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134832 4918 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:57.149924 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134835 4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:57.149924 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134838 4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:57.149924 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134840 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:57.149924 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134843 4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:57.149924 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134845 4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:57.149924 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134848 4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:57.149924 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134850 4918 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:57.149924 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134853 4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:57.149924 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134855 4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:57.149924 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134858 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:57.149924 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134860 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:57.149924 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134864 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:57.149924 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134867 4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:57.149924 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134869 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:57.150426 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134872 4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:57.150426 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134874 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:57.150426 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134877 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:57.150426 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134881 4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:57.150426 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134883 4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:57.150426 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134886 4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:57.150426 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.134888 4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:57.150426 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.138544 4918 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 19:56:57.150426 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.138565 4918 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 19:56:57.150426 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.138584 4918 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 19:56:57.150426 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.138589 4918 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 19:56:57.150426 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138646 4918 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:57.150426 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138651 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:57.150426 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138654 4918 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:57.150426 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138657 4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:57.150426 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138660 4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:57.150426 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138663 4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:57.150426 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138665 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:57.150426 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138668 4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:57.150912 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138671 4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:57.150912 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138674 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:57.150912 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138676 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:57.150912 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138679 4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:57.150912 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138682 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:57.150912 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138685 4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:57.150912 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138688 4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:57.150912 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138699 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:57.150912 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138703 4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:57.150912 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138706 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:57.150912 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138709 4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:57.150912 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138712 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:57.150912 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138715 4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:57.150912 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138717 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:57.150912 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138720 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:57.150912 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138723 4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:57.150912 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138725 4918 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:57.150912 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138728 4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:57.150912 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138730 4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:57.151416 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138733 4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:57.151416 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138736 4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:57.151416 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138739 4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:57.151416 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138742 4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:57.151416 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138744 4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:57.151416 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138747 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:57.151416 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138750 4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:57.151416 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138752 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:57.151416 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138755 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:57.151416 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138757 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:57.151416 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138761 4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:57.151416 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138764 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:57.151416 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138767 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:57.151416 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138769 4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:57.151416 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138772 4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:57.151416 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138775 4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:57.151416 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138777 4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:57.151416 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138780 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:57.151416 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138783 4918 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:57.151882 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138786 4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:57.151882 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138788 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:57.151882 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138791 4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:57.151882 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138794 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:57.151882 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138796 4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:57.151882 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138799 4918 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:57.151882 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138801 4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:57.151882 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138804 4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:57.151882 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138807 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:57.151882 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138809 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:57.151882 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138812 4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:57.151882 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138814 4918 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:57.151882 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138817 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:57.151882 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138821 4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:57.151882 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138825 4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:57.151882 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138828 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:57.151882 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138832 4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:57.151882 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138836 4918 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:57.151882 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138840 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:57.152368 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138843 4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:57.152368 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138845 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:57.152368 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138848 4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:57.152368 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138851 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:57.152368 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138854 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:57.152368 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138857 4918 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:57.152368 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138860 4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:57.152368 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138862 4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:57.152368 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138865 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:57.152368 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138868 4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:57.152368 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138871 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:57.152368 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138874 4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:57.152368 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138876 4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:57.152368 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138879 4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:57.152368 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138882 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:57.152368 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138885 4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:57.152368 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138887 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:57.152368 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138890 4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:57.152368 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138893 4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:57.152870 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138895 4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:57.152870 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138898 4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:57.152870 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138900 4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:57.152870 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138903 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:57.152870 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138906 4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:57.152870 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138909 4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:57.152870 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138911 4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:57.152870 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138914 4918 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:57.152870 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138916 4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:57.152870 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138919 4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:57.152870 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138921 4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:57.152870 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138925 4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:57.152870 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138928 4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:57.152870 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138931 4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:57.152870 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138934 4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:57.152870 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138937 4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:57.152870 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138940 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:57.152870 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138943 4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:57.152870 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138946 4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:57.152870 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138948 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:57.153404 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138951 4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:57.153404 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.138954 4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:57.153404 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.138981 4918 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:56:57.153404 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139118 4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:57.153404 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139122 4918 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:57.153404 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139126 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:57.153404 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139129 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:57.153404 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139132 4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:57.153404 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139135 4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:57.153404 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139138 4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:57.153404 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139141 4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:57.153404 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139144 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:57.153404 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139146 4918 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:57.153404 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139149 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:57.153404 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139152 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:57.153404 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139154 4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:57.153404 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139157 4918 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:57.153810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139160 4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:57.153810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139162 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:57.153810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139165 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:57.153810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139168 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:57.153810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139171 4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:57.153810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139187 4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:57.153810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139191 4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:57.153810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139196 4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:57.153810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139199 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:57.153810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139203 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:57.153810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139205 4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:57.153810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139208 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:57.153810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139212 4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:57.153810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139216 4918 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:57.153810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139219 4918 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:57.153810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139222 4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:57.153810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139225 4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:57.153810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139228 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:57.153810 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139230 4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:57.154280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139233 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:57.154280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139235 4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:57.154280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139238 4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:57.154280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139241 4918 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:57.154280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139243 4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:57.154280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139246 4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:57.154280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139249 4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:57.154280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139251 4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:57.154280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139254 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:57.154280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139256 4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:57.154280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139259 4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:57.154280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139261 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:57.154280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139264 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:57.154280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139266 4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:57.154280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139269 4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:57.154280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139272 4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:57.154280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139274 4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:57.154280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139277 4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:57.154280 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139279 4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:57.154745 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139282 4918 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:57.154745 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139284 4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:57.154745 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139287 4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:57.154745 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139290 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:57.154745 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139293 4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:57.154745 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139296 4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:57.154745 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139298 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:57.154745 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139301 4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:57.154745 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139304 4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:57.154745 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139306 4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:57.154745 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139309 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:57.154745 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139311 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:57.154745 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139314 4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:57.154745 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139317 4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:57.154745 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139319 4918 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:57.154745 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139322 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:57.154745 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139324 4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:57.154745 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139327 4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:57.154745 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139329 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:57.154745 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139332 4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:57.155312 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139334 4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:57.155312 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139337 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:57.155312 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139339 4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:57.155312 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139342 4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:57.155312 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139345 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:57.155312 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139348 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:57.155312 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139350 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:57.155312 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139353 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:57.155312 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139355 4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:57.155312 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139359 4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:57.155312 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139363 4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:57.155312 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139366 4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:57.155312 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139369 4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:57.155312 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139371 4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:57.155312 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139374 4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:57.155312 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139382 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:57.155312 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139385 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:57.155312 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139387 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:57.155312 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139393 4918 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:57.155312 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139395 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:57.155814 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139398 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:57.155814 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139401 4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:57.155814 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139403 4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:57.155814 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139406 4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:57.155814 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139408 4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:57.155814 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139411 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:57.155814 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139413 4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:57.155814 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139416 4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:57.155814 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139418 4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:57.155814 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139421 4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:57.155814 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139424 4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:57.155814 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139426 4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:57.155814 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139429 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:57.155814 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139432 4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:57.155814 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.139456 4918 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:56:57.155814 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139490 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:57.156323 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139494 4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:57.156323 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139497 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:57.156323 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139500 4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:57.156323 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139503 4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:57.156323 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139506 4918 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:57.156323 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139508 4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:57.156323 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139511 4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:57.156323 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139514 4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:57.156323 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139516 4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:57.156323 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139519 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:57.156323 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139522 4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:57.156323 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139524 4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:57.156323 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139527 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:57.156323 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139530 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:57.156323 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139534 4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:57.156323 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139537 4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:57.156323 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139539 4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:57.156323 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139542 4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:57.156323 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139545 4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:57.156833 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139547 4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:57.156833 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139550 4918 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:57.156833 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139552 4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:57.156833 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139557 4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:57.156833 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139559 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:57.156833 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139562 4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:57.156833 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139564 4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:57.156833 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139567 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:57.156833 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139570 4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:57.156833 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139572 4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:57.156833 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139575 4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:57.156833 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139578 4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:57.156833 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139580 4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:57.156833 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139583 4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:57.156833 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139585 4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:57.156833 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139588 4918 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:57.156833 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139591 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:57.156833 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139594 4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:57.156833 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139596 4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:57.156833 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139599 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:57.157395 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139602 4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:57.157395 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139604 4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:57.157395 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139607 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:57.157395 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139609 4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:57.157395 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139612 4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:57.157395 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139614 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:57.157395 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139618 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:57.157395 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139622 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:57.157395 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139624 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:57.157395 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139627 4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:57.157395 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139631 4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:57.157395 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139635 4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:57.157395 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139638 4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:57.157395 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139641 4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:57.157395 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139644 4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:57.157395 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139648 4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:57.157395 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139651 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:57.157395 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139653 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:57.157395 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139656 4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:57.157395 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139659 4918 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:57.157875 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139661 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:57.157875 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139664 4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:57.157875 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139667 4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:57.157875 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139669 4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:57.157875 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139672 4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:57.157875 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139674 4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:57.157875 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139677 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:57.157875 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139680 4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:57.157875 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139682 4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:57.157875 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139685 4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:57.157875 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139687 4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:57.157875 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139690 4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:57.157875 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139692 4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:57.157875 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139695 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:57.157875 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139698 4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:57.157875 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139700 4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:57.157875 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139703 4918 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:57.157875 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139705 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:57.157875 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139708 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:57.157875 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139712 4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:57.158379 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139715 4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:57.158379 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139718 4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:57.158379 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139720 4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:57.158379 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139723 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:57.158379 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139726 4918 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:57.158379 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139728 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:57.158379 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139731 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:57.158379 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139734 4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:57.158379 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139737 4918 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:57.158379 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139740 4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:57.158379 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139743 4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:57.158379 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139745 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:57.158379 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139748 4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:57.158379 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139750 4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:57.158379 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139753 4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:57.158379 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139756 4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:57.158379 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139758 4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:57.158379 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139761 4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:57.158379 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139763 4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:57.159084 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139766 4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:57.159084 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139769 4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:57.159084 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139773 4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:57.159084 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139776 4918 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:57.159084 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139779 4918 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:57.159084 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139782 4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:57.159084 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:56:57.139784 4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:57.159084 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.139834 4918 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 19:56:57.159084 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.140783 4918 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 19:56:57.159084 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.140831 4918 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 19:56:57.159084 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.141017 4918 server.go:1014] "Starting client certificate rotation" Nov 03 19:56:57.159084 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.141152 4918 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:56:57.159084 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.141221 4918 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:51:15 +0000 UTC" deadline="2028-07-10 22:00:43.076845693 +0000 UTC" Nov 03 19:56:57.159084 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.141251 4918 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14762h3m45.935597892s" Nov 03 19:56:57.159084 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.141443 4918 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:56:57.159626 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.142456 4918 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:56:57.159626 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.145470 4918 log.go:25] "Validated CRI v1 runtime API" Nov 03 19:56:57.159626 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.147139 4918 log.go:25] "Validated CRI v1 image API" Nov 03 19:56:57.159626 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.147834 4918 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 19:56:57.159626 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.149681 4918 fs.go:135] Filesystem UUIDs: map[43ddb09a-1a9b-4f99-9cb1-075c06fcec75:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a5e5e004-9e73-4ed8-91ad-730ff35b9edf:/dev/nvme0n1p3] Nov 03 19:56:57.159808 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.149695 4918 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00cd428a9ec79e829670ad6140c6754ad0c64d61c98eff02f9bbfed9a43eb089/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00cd428a9ec79e829670ad6140c6754ad0c64d61c98eff02f9bbfed9a43eb089/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/059ace63cca414c0b2cfdff9df65ba6bf009d06d374a5a43d5fc33edeb68c7d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/059ace63cca414c0b2cfdff9df65ba6bf009d06d374a5a43d5fc33edeb68c7d9/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08ece5f0f14b1d4f47b31345c61311442bebb290188fbfdbb16987fa5037f038/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08ece5f0f14b1d4f47b31345c61311442bebb290188fbfdbb16987fa5037f038/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ca9416f4f2401864659a5a0f7debfb2d9b3f44c8f9b1ff02a8fa40dca952bb5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ca9416f4f2401864659a5a0f7debfb2d9b3f44c8f9b1ff02a8fa40dca952bb5/userdata/shm major:0 minor:323 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26e2d32ed3f80f30a74bf7ab2ae1ed4231422c16e0856fa91e22aaf14e1500f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26e2d32ed3f80f30a74bf7ab2ae1ed4231422c16e0856fa91e22aaf14e1500f8/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3451bfcb76a5e9ccc307b0620ef67625c59780543a4204f331a91b78057599c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3451bfcb76a5e9ccc307b0620ef67625c59780543a4204f331a91b78057599c7/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5440d31e29ee75e3c31b392353d75842f76a00ab037294e19e4faa80f7f22d33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5440d31e29ee75e3c31b392353d75842f76a00ab037294e19e4faa80f7f22d33/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/676071769d6a0e543d6500b74fd35bf48763179597301db835c32ceb21c72e95/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/676071769d6a0e543d6500b74fd35bf48763179597301db835c32ceb21c72e95/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6accded0538df0201232b8f979dadc1809816fb77ad11794fc7c2dfdcf04087d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6accded0538df0201232b8f979dadc1809816fb77ad11794fc7c2dfdcf04087d/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7149a3ee7166a852cfec10096e691606bf8c920236a0a5a1c61f6d7330c237cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7149a3ee7166a852cfec10096e691606bf8c920236a0a5a1c61f6d7330c237cf/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72b25779bfb194395d3b79bb0ba2f8fc2ffe4b0e3b3f2dda6d6a6c02308c0c8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72b25779bfb194395d3b79bb0ba2f8fc2ffe4b0e3b3f2dda6d6a6c02308c0c8b/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/75a18ed8c31658475f68e433ccb3231e073366526648ddee635eba9cbe17ef84/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/75a18ed8c31658475f68e433ccb3231e073366526648ddee635eba9cbe17ef84/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7850e679b9b2db944c0e6df47944040d848ca279a5b6f526b886319f5a6747e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7850e679b9b2db944c0e6df47944040d848ca279a5b6f526b886319f5a6747e1/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80dc414f095cf57e0972d1f3e55a667bde76a7616d344e0ea893b46227440ef1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80dc414f095cf57e0972d1f3e55a667bde76a7616d344e0ea893b46227440ef1/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/893a64eb71fdd26145f701505822dc8f1b865079010077d13e0ab2bc24fbfbcd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/893a64eb71fdd26145f701505822dc8f1b865079010077d13e0ab2bc24fbfbcd/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d07cd3031c2ed29009a1a485c0dbd36bbb170b5754e25889aaa3d1eb9607e47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d07cd3031c2ed29009a1a485c0dbd36bbb170b5754e25889aaa3d1eb9607e47/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/affcf81e798d485c91871d4746b1d40e88faf886046aec06b260249516ab23cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/affcf81e798d485c91871d4746b1d40e88faf886046aec06b260249516ab23cf/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5d1ca779ca687573ee12ef853bfbf661e111e416c73c9103ef33a849cad0439/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5d1ca779ca687573ee12ef853bfbf661e111e416c73c9103ef33a849cad0439/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0def15a10438ae80cc43adfc56c708f394bb9e93cfd290dd1c406f282a41777/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0def15a10438ae80cc43adfc56c708f394bb9e93cfd290dd1c406f282a41777/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da3861e8abc8d28dfb517bc237f42ec79e979910ae486a1c3c1e93c54d392277/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da3861e8abc8d28dfb517bc237f42ec79e979910ae486a1c3c1e93c54d392277/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6088cca271321bf7ced864ab2bb5e69a37f875ca07627db935442ce2ebb1694/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6088cca271321bf7ced864ab2bb5e69a37f875ca07627db935442ce2ebb1694/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6294d9eff9fe195712f394713efe6f81e7a0d46bcbb1b544a7c758481901a3c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6294d9eff9fe195712f394713efe6f81e7a0d46bcbb1b544a7c758481901a3c/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc26bdf4cad8d170f3748ba39a20387db0e062e58189b5bb3753406739d9bf26/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc26bdf4cad8d170f3748ba39a20387db0e062e58189b5bb3753406739d9bf26/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d9796ba-d26b-494e-a4b1-2ebab6496f49/volumes/kubernetes.io~projected/kube-api-access-pgswz:{mountpoint:/var/lib/kubelet/pods/0d9796ba-d26b-494e-a4b1-2ebab6496f49/volumes/kubernetes.io~projected/kube-api-access-pgswz major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/158d8c01-2d31-43b4-a098-0c731c1fd163/volumes/kubernetes.io~projected/kube-api-access-wl89h:{mountpoint:/var/lib/kubelet/pods/158d8c01-2d31-43b4-a098-0c731c1fd163/volumes/kubernetes.io~projected/kube-api-access-wl89h major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fc8cec5-9248-4a47-822e-d81b37a380ef/volumes/kubernetes.io~projected/kube-api-access-kzdnd:{mountpoint:/var/lib/kubelet/pods/1fc8cec5-9248-4a47-822e-d81b37a380ef/volumes/kubernetes.io~projected/kube-api-access-kzdnd major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44/volumes/kubernetes.io~projected/kube-api-access-6hb8f:{mountpoint:/var/lib/kubelet/pods/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44/volumes/kubernetes.io~projected/kube-api-access-6hb8f major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26ac4e9f-3445-4994-aa1b-086b39b8c1cc/volumes/kubernetes.io~projected/kube-api-access-7bqcz:{mountpoint:/var/lib/kubelet/pods/26ac4e9f-3445-4994-aa1b-086b39b8c1cc/volumes/kubernetes.io~projected/kube-api-access-7bqcz major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33b6188c-0202-4bc8-813f-2d189587f3aa/volumes/kubernetes.io~projected/kube-api-access-gxblp:{mountpoint:/var/lib/kubelet/pods/33b6188c-0202-4bc8-813f-2d189587f3aa/volumes/kubernetes.io~projected/kube-api-access-gxblp major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/433c5b6b-176b-44d1-afc0-6884045e468c/volumes/kubernetes.io~projected/kube-api-access-8x4lc:{mountpoint:/var/lib/kubelet/pods/433c5b6b-176b-44d1-afc0-6884045e468c/volumes/kubernetes.io~projected/kube-api-access-8x4lc major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/433c5b6b-176b-44d1-afc0-6884045e468c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/433c5b6b-176b-44d1-afc0-6884045e468c/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/449a05ee-ead2-4507-8d65-faaeb3ab3281/volumes/kubernetes.io~projected/kube-api-access-zxkg4:{mountpoint:/var/lib/kubelet/pods/449a05ee-ead2-4507-8d65-faaeb3ab3281/volumes/kubernetes.io~projected/kube-api-access-zxkg4 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/449a05ee-ead2-4507-8d65-faaeb3ab3281/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/449a05ee-ead2-4507-8d65-faaeb3ab3281/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4726eb4c-48e8-423d-a62c-250a18bd8b83/volumes/kubernetes.io~projected/kube-api-access-ml5nd:{mountpoint:/var/lib/kubelet/pods/4726eb4c-48e8-423d-a62c-250a18bd8b83/volumes/kubernetes.io~projected/kube-api-access-ml5nd major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/514e4bfb-77ef-4b1f-977a-3f9308cf886c/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/514e4bfb-77ef-4b1f-977a-3f9308cf886c/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/514e4bfb-77ef-4b1f-977a-3f9308cf886c/volumes/kubernetes.io~projected/kube-api-access-djg6c:{mountpoint:/var/lib/kubelet/pods/514e4bfb-77ef-4b1f-977a-3f9308cf886c/volumes/kubernetes.io~projected/kube-api-access-djg6c major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/514e4bfb-77ef-4b1f-977a-3f9308cf886c/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/514e4bfb-77ef-4b1f-977a-3f9308cf886c/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/514e4bfb-77ef-4b1f-977a-3f9308cf886c/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/514e4bfb-77ef-4b1f-977a-3f9308cf886c/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a7d187d-9a17-475b-8ee4-ae5836d9a73a/volumes/kubernetes.io~projected/kube-api-access-gkf49:{mountpoint:/var/lib/kubelet/pods/5a7d187d-9a17-475b-8ee4-ae5836d9a73a/volumes/kubernetes.io~projected/kube-api-access-gkf49 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61f1d7a9-a8b3-4db5-878d-4b8d1632fafc/volumes/kubernetes.io~projected/kube-api-access-jh8xj:{mountpoint:/var/lib/kubelet/pods/61f1d7a9-a8b3-4db5-878d-4b8d1632fafc/volumes/kubernetes.io~projected/kube-api-access-jh8xj major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/653cb377-919b-4793-85a4-0d60d6aeaff6/volumes/kubernetes.io~projected/kube-api-access-vb6j8:{mountpoint:/var/lib/kubelet/pods/653cb377-919b-4793-85a4-0d60d6aeaff6/volumes/kubernetes.io~projected/kube-api-access-vb6j8 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/653cb377-919b-4793-85a4-0d60d6aeaff6/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/653cb377-919b-4793-85a4-0d60d6aeaff6/volumes/kubernetes.io~secret/default-certificate major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/653cb377-919b-4793-85a4-0d60d6aeaff6/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/653cb377-919b-4793-85a4-0d60d6aeaff6/volumes/kubernetes.io~secret/stats-auth major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6de4fd5d-d10a-4630-80bf-3838fb02d679/volumes/kubernetes.io~projected/kube-api-access-8f594:{mountpoint:/var/lib/kubelet/pods/6de4fd5d-d10a-4630-80bf-3838fb02d679/volumes/kubernetes.io~projected/kube-api-access-8f594 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40/volumes/kubernetes.io~projected/kube-api-access-m7vhv:{mountpoint:/var/lib/kubelet/pods/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40/volumes/kubernetes.io~projected/kube-api-access-m7vhv major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88c8909b-9acd-46d0-9f19-d74e2db51f77/volumes/kubernetes.io~projected/kube-api-access-7g8dn:{mountpoint:/var/lib/kubelet/pods/88c8909b-9acd-46d0-9f19-d74e2db51f77/volumes/kubernetes.io~projected/kube-api-access-7g8dn major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f/volumes/kubernetes.io~projected/kube-api-access-dd49x:{mountpoint:/var/lib/kubelet/pods/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f/volumes/kubernetes.io~projected/kube-api-access-dd49x major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96087edc-f246-4ba4-a9d6-734a997da1f9/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/96087edc-f246-4ba4-a9d6-734a997da1f9/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99713fae-ae74-4da5-b435-c73edc1c4843/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/99713fae-ae74-4da5-b435-c73edc1c4843/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99713fae-ae74-4da5-b435-c73edc1c4843/volumes/kubernetes.io~projected/kube-api-access-ctj28:{mountpoint:/var/lib/kubelet/pods/99713fae-ae74-4da5-b435-c73edc1c4843/volumes/kubernetes.io~projected/kube-api-access-ctj28 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99713fae-ae74-4da5-b435-c73edc1c4843/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/99713fae-ae74-4da5-b435-c73edc1c4843/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a24da6cd-d805-4f8a-ac99-89dbd45c86c3/volumes/kubernetes.io~projected/kube-api-access-5jhkj:{mountpoint:/var/lib/kubelet/pods/a24da6cd-d805-4f8a-ac99-89dbd45c86c3/volumes/kubernetes.io~projected/kube-api-access-5jhkj major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab34f1da-ace1-443d-bddd-9d9905d22e52/volumes/kubernetes.io~projected/kube-api-access-brzfd:{mountpoint:/var/lib/kubelet/pods/ab34f1da-ace1-443d-bddd-9d9905d22e52/volumes/kubernetes.io~projected/kube-api-access-brzfd major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adc4cee9-00f0-4fac-b655-f58c21377362/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/adc4cee9-00f0-4fac-b655-f58c21377362/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adc4cee9-00f0-4fac-b655-f58c21377362/volumes/kubernetes.io~projected/kube-api-access-6sx9r:{mountpoint:/var/lib/kubelet/pods/adc4cee9-00f0-4fac-b655-f58c21377362/volumes/kubernetes.io~projected/kube-api-access-6sx9r major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adc4cee9-00f0-4fac-b655-f58c21377362/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/adc4cee9-00f0-4fac-b655-f58c21377362/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adc4cee9-00f0-4fac-b655-f58c21377362/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/adc4cee9-00f0-4fac-b655-f58c21377362/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aed0ee7d-d648-4138-b5ef-2ae6795321d2/volumes/kubernetes.io~projected/kube-api-access-w5sh5:{mountpoint:/var/lib/kubelet/pods/aed0ee7d-d648-4138-b5ef-2ae6795321d2/volumes/kubernetes.io~projected/kube-api-access-w5sh5 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aed0ee7d-d648-4138-b5ef-2ae6795321d2/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/aed0ee7d-d648-4138-b5ef-2ae6795321d2/volumes/kubernetes.io~secret/cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c60cb950-5dd9-4a2b-bba3-b3c38d3591e3/volumes/kubernetes.io~projected/kube-api-access-6vgq9:{mountpoint:/var/lib/kubelet/pods/c60cb950-5dd9-4a2b-bba3-b3c38d3591e3/volumes/kubernetes.io~projected/kube-api-access-6vgq9 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2e4493e-d29f-49b6-851e-cbd01214bf5c/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d2e4493e-d29f-49b6-851e-cbd01214bf5c/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7dde7dd-245b-4437-9464-0e8b739caa98/volumes/kubernetes.io~projected/kube-api-access-z6ggm:{mountpoint:/var/lib/kubelet/pods/f7dde7dd-245b-4437-9464-0e8b739caa98/volumes/kubernetes.io~projected/kube-api-access-z6ggm major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f838cffc-2fc7-4a3f-880b-8a80298cb1b4/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f838cffc-2fc7-4a3f-880b-8a80298cb1b4/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f838cffc-2fc7-4a3f-880b-8a80298cb1b4/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f838cffc-2fc7-4a3f-880b-8a80298cb1b4/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f838cffc-2fc7-4a3f-880b-8a80298cb1b4/volumes/kubernetes.io~projected/kube-api-access-sk6q8:{mountpoint:/var/lib/kubelet/pods/f838cffc-2fc7-4a3f-880b-8a80298cb1b4/volumes/kubernetes.io~projected/kube-api-access-sk6q8 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f90a0a4e-2794-46fc-a799-9c1593d859ab/volumes/kubernetes.io~projected/kube-api-access-tsb75:{mountpoint:/var/lib/kubelet/pods/f90a0a4e-2794-46fc-a799-9c1593d859ab/volumes/kubernetes.io~projected/kube-api-access-tsb75 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79/volumes/kubernetes.io~projected/kube-api-access-gtdbf:{mountpoint:/var/lib/kubelet/pods/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79/volumes/kubernetes.io~projected/kube-api-access-gtdbf major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff051e4a-95a6-4c0d-9f69-773974d6eb7c/volumes/kubernetes.io~projected/kube-api-access-gm2tl:{mountpoint:/var/lib/kubelet/pods/ff051e4a-95a6-4c0d-9f69-773974d6eb7c/volumes/kubernetes.io~projected/kube-api-access-gm2tl major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff051e4a-95a6-4c0d-9f69-773974d6eb7c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ff051e4a-95a6-4c0d-9f69-773974d6eb7c/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/d0a0d627c663ecfa7c9aa2f4739584ff624e7d810fc7babc4af2e99173af6b05/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b0a615ec031c34f56fad4e987a0a55f5c21b4c8e52e6f5bd9c0ea21694172ee6/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ab22571401e220ac588ebcc49003cf1fadee59f8971cc54b1040b23b1082c5e8/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/875f6b84ad25546fbed92c34905d26ffee969b4c733fcfb7f16977f08c84d105/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/6b3c3b7cce48f54033ddbab0cbc7e63992397c1efff19d7229399132329a4247/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/f76f1d8e7ef585bf193618f3f5dca87a4b4988293d486ce832382e7059265081/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/1d089e4931c3881a41e553d7b25e9e47b8de113dfe1b0e7a1bd3a800df8910a1/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/24e750f6c9c3df0c6521e9d53589b8487d02e42f947cdd1e2315d78123fad454/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/48994b21940c4dbaa263ec37a586f292aadc8ffdff13060b2efa538e0c621031/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/c54a3cf7ab7f38b169d46f834db2c17a07dece1f94ea122de906d450fae7507e/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/b2713dbf0977fbf8593c9ef4a3fde1562ad471e49af59afc3d822c497c22a23b/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/d8613f29135bcb2deb464b4c5bfbbdf0d95907f74ae9c47a38c565402ca4d939/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/d5ced45e450551da5ff00e73740fa49aeb794f6ece72cede43799d0bd86df7db/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/1328e7cc5c8a8646f051983e37b490233f70b6a175f0ff72af6217834b795c0d/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/9cc64215828174f6a59d6b6499fa5f8bd63f2b686bcfe50c4924b3c91489b911/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/2de3edef4628d5ae4dcdaf965f0385d21cb8de02f8c155787728e4a6e5439a39/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/4e3b2be5f9c14c0763585368bc9dae9be8d78f53863fa774f175d9d22396128e/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/05c8a698a4ff0d6a2920debc9de7713508029c69863d393a47ee78139d11068e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/81c3a63b426ce145b3ecdabcae8ee7b27d63159cc52583cb2bbf30d677d497d8/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/3b93888aaa95117114bf588398c12e56279ad84a31addb33371ca37733218847/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/40fb07371e71cd6a7588bd21ab060b3db5e68533d6bce8fabf9170b64239d93f/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/a1d6de818148a675fdbf3cd5c0e2a28919bbd9d4e3bc780f7737893fd5226edf/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/6335c99c066f0b7debd6861f2a6022d26b8132ce56f18c9635c8c952e3715e0e/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/4a2373156b240b52df991ca12610b8705a86d0b24ff6382265406a42d9cb9b86/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/f5c679d72d2bd33a3606999f98e3f3f5ef17aec226f2c3a941f65fd9cadd7885/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/e0d52ef658f593036d24d06004dcb0e2957ae7902d5a42606459a916a987cc4e/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/201c40bba77e7d00bd6180b5d753b45f64583ecf7c6e9daa2aad8fa0bd9f9e27/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/d8e9e4ce3d243c919143f6a7f9b249be67527f7a747850718e01f5c3de8e1d8a/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/3f8543a7032ffb203e28c1c0e85e8d43e1811635e92e6485247dcac29f8ef0ca/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/948e7c8ef03e055765a54a8abe61ad90994e86079d35bd78108a3425bdf60767/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/6e76906d0069956d920c8625c8902f620b8b6eaff2a6e8490bc8eb3fa0fc97cc/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/49b238a5fe0e25e02b943602cd1e4d507c9f21c56534f736093edba101345ff5/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/1ba5bc199e3b7f36b3aada7ac82d034f2e426fd8e6b668c8a95343b42595b8b4/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/3cf0ee2e4f502a67d50f1b1b76d475af10677ecb79878fd235268b659321449e/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/5a1a08b02fd9e88601adaae3d867b7acd2aa777a8dec5e5e74c8bf86ff396250/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/bad7d6ca8b14b224db024c2bc0336a53eff8ad8b880533fcb04de35a01b406b1/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/0ee1495e8a410858d1b4cd139baa4d3a2744f5e9cb4084a6e12329f0994aba5b/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/03570a4ce61f0a1aa3b1c57465ef409ac23aae0a6e206eaa7690f2660b4f3057/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/16ba4e9e3579e12043f2509b815719f52847fc71cda3f2c03a2a84dd0ee041d0/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/dc09b44b38a06f9981b9a23536228fcb49ce55df6469d39ccf9af531fd0c3b0b/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/aeb03b8ecceb128849677af726c753d30efe3a40be24f3a59a411a8b06872517/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/c689a93d90b28accc4fdda2ab4e0caa0d314d0c3cc826a6f20764a8722a4e80d/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/d33fa55d4ca6f51b89677bbec104be2d785853bd1dcd3d232cf65de07114e5fa/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/574ed17d921c97d0d2fb8899f74def89d45ca1c6c1cc0d99cd8e677553e121ec/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/3fa27daf408045432ad1256bcee3b6c15f98c6d71dd190ac83345ba220cb36a1/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/bab6b9cad2d1aec7019ba1822d3533216025eab02bf43c5f7bf09be09692ab31/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/747f494d6f0a563aa4577b9d97063ca7c9e70d72b7a7e617363e00d4a1cf856b/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/294f899681f46a3a3b025ae2d295b4e37113dd2968283e69e37810cbcf46244b/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/1298f5d8cb248bc83ec990aff301198750a2b520d2fcf2a846b8f9cc769e7d1e/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/e79577e480de9f1565ff239bf1ef226a8812fda2c8f73d92639be57cc98bf40e/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/06c6e76913c71071bbeba0920e0e245aad5b9ab68a18dac332f6952071f51ef1/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/e8d50863458f5cd370d238ef14157c9f778061f7b452d7b84d2e9fea9f837cdd/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/0e29fca16cd2be4d56945b3ba6de0646a3212b41b89c02d9ddf0ddd764ae74bb/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/1633c525d5b4638e0dfb5bc91cf3d6cb8ec40c72dacf5cc703a209a22d0e4119/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/f9d95fab49272491e2bb69f9c0ba1beeba4bc7a0301954069fdd1c7229e33c3a/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/3fa1b4cb54c9f315583b569cfd6e6840a4ffd17f83b5249388f3ed5affc9aa66/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 19:56:57.160083 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.157147 4918 manager.go:217] Machine: {Timestamp:2026-11-03 19:56:57.156667835 +0000 UTC m=+0.088535338 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109759 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec282771c9f1812d2fbdc58594c4299e SystemUUID:ec282771-c9f1-812d-2fbd-c58594c4299e BootID:e3331871-b391-470d-8867-3aa693befd9c Filesystems:[{Device:/run/containers/storage/overlay-containers/f6294d9eff9fe195712f394713efe6f81e7a0d46bcbb1b544a7c758481901a3c/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/affcf81e798d485c91871d4746b1d40e88faf886046aec06b260249516ab23cf/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da3861e8abc8d28dfb517bc237f42ec79e979910ae486a1c3c1e93c54d392277/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00cd428a9ec79e829670ad6140c6754ad0c64d61c98eff02f9bbfed9a43eb089/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ff051e4a-95a6-4c0d-9f69-773974d6eb7c/volumes/kubernetes.io~projected/kube-api-access-gm2tl DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/449a05ee-ead2-4507-8d65-faaeb3ab3281/volumes/kubernetes.io~projected/kube-api-access-zxkg4 DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3451bfcb76a5e9ccc307b0620ef67625c59780543a4204f331a91b78057599c7/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c60cb950-5dd9-4a2b-bba3-b3c38d3591e3/volumes/kubernetes.io~projected/kube-api-access-6vgq9 DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f90a0a4e-2794-46fc-a799-9c1593d859ab/volumes/kubernetes.io~projected/kube-api-access-tsb75 DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f838cffc-2fc7-4a3f-880b-8a80298cb1b4/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/653cb377-919b-4793-85a4-0d60d6aeaff6/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/adc4cee9-00f0-4fac-b655-f58c21377362/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79/volumes/kubernetes.io~projected/kube-api-access-gtdbf DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ca9416f4f2401864659a5a0f7debfb2d9b3f44c8f9b1ff02a8fa40dca952bb5/userdata/shm DeviceMajor:0 DeviceMinor:323 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d07cd3031c2ed29009a1a485c0dbd36bbb170b5754e25889aaa3d1eb9607e47/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/676071769d6a0e543d6500b74fd35bf48763179597301db835c32ceb21c72e95/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/653cb377-919b-4793-85a4-0d60d6aeaff6/volumes/kubernetes.io~projected/kube-api-access-vb6j8 DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/33b6188c-0202-4bc8-813f-2d189587f3aa/volumes/kubernetes.io~projected/kube-api-access-gxblp DeviceMajor:0 DeviceMinor:310 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/61f1d7a9-a8b3-4db5-878d-4b8d1632fafc/volumes/kubernetes.io~projected/kube-api-access-jh8xj DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f/volumes/kubernetes.io~projected/kube-api-access-dd49x DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99713fae-ae74-4da5-b435-c73edc1c4843/volumes/kubernetes.io~projected/kube-api-access-ctj28 DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08ece5f0f14b1d4f47b31345c61311442bebb290188fbfdbb16987fa5037f038/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a24da6cd-d805-4f8a-ac99-89dbd45c86c3/volumes/kubernetes.io~projected/kube-api-access-5jhkj DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/449a05ee-ead2-4507-8d65-faaeb3ab3281/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/adc4cee9-00f0-4fac-b655-f58c21377362/volumes/kubernetes.io~projected/kube-api-access-6sx9r DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0def15a10438ae80cc43adfc56c708f394bb9e93cfd290dd1c406f282a41777/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff051e4a-95a6-4c0d-9f69-773974d6eb7c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4726eb4c-48e8-423d-a62c-250a18bd8b83/volumes/kubernetes.io~projected/kube-api-access-ml5nd DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/99713fae-ae74-4da5-b435-c73edc1c4843/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6de4fd5d-d10a-4630-80bf-3838fb02d679/volumes/kubernetes.io~projected/kube-api-access-8f594 DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/433c5b6b-176b-44d1-afc0-6884045e468c/volumes/kubernetes.io~projected/kube-api-access-8x4lc DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40/volumes/kubernetes.io~projected/kube-api-access-m7vhv DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/514e4bfb-77ef-4b1f-977a-3f9308cf886c/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/158d8c01-2d31-43b4-a098-0c731c1fd163/volumes/kubernetes.io~projected/kube-api-access-wl89h DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adc4cee9-00f0-4fac-b655-f58c21377362/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/514e4bfb-77ef-4b1f-977a-3f9308cf886c/volumes/kubernetes.io~projected/kube-api-access-djg6c DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5a7d187d-9a17-475b-8ee4-ae5836d9a73a/volumes/kubernetes.io~projected/kube-api-access-gkf49 DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f838cffc-2fc7-4a3f-880b-8a80298cb1b4/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/514e4bfb-77ef-4b1f-977a-3f9308cf886c/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aed0ee7d-d648-4138-b5ef-2ae6795321d2/volumes/kubernetes.io~projected/kube-api-access-w5sh5 DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc26bdf4cad8d170f3748ba39a20387db0e062e58189b5bb3753406739d9bf26/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6088cca271321bf7ced864ab2bb5e69a37f875ca07627db935442ce2ebb1694/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1fc8cec5-9248-4a47-822e-d81b37a380ef/volumes/kubernetes.io~projected/kube-api-access-kzdnd DeviceMajor:0 DeviceMinor:333 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/26ac4e9f-3445-4994-aa1b-086b39b8c1cc/volumes/kubernetes.io~projected/kube-api-access-7bqcz DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7149a3ee7166a852cfec10096e691606bf8c920236a0a5a1c61f6d7330c237cf/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26e2d32ed3f80f30a74bf7ab2ae1ed4231422c16e0856fa91e22aaf14e1500f8/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5440d31e29ee75e3c31b392353d75842f76a00ab037294e19e4faa80f7f22d33/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/88c8909b-9acd-46d0-9f19-d74e2db51f77/volumes/kubernetes.io~projected/kube-api-access-7g8dn DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72b25779bfb194395d3b79bb0ba2f8fc2ffe4b0e3b3f2dda6d6a6c02308c0c8b/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/653cb377-919b-4793-85a4-0d60d6aeaff6/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44/volumes/kubernetes.io~projected/kube-api-access-6hb8f DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/893a64eb71fdd26145f701505822dc8f1b865079010077d13e0ab2bc24fbfbcd/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6accded0538df0201232b8f979dadc1809816fb77ad11794fc7c2dfdcf04087d/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab34f1da-ace1-443d-bddd-9d9905d22e52/volumes/kubernetes.io~projected/kube-api-access-brzfd DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7850e679b9b2db944c0e6df47944040d848ca279a5b6f526b886319f5a6747e1/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/75a18ed8c31658475f68e433ccb3231e073366526648ddee635eba9cbe17ef84/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/96087edc-f246-4ba4-a9d6-734a997da1f9/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f838cffc-2fc7-4a3f-880b-8a80298cb1b4/volumes/kubernetes.io~projected/kube-api-access-sk6q8 DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99713fae-ae74-4da5-b435-c73edc1c4843/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5d1ca779ca687573ee12ef853bfbf661e111e416c73c9103ef33a849cad0439/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adc4cee9-00f0-4fac-b655-f58c21377362/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7dde7dd-245b-4437-9464-0e8b739caa98/volumes/kubernetes.io~projected/kube-api-access-z6ggm DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aed0ee7d-d648-4138-b5ef-2ae6795321d2/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d2e4493e-d29f-49b6-851e-cbd01214bf5c/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d9796ba-d26b-494e-a4b1-2ebab6496f49/volumes/kubernetes.io~projected/kube-api-access-pgswz DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/514e4bfb-77ef-4b1f-977a-3f9308cf886c/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/433c5b6b-176b-44d1-afc0-6884045e468c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80dc414f095cf57e0972d1f3e55a667bde76a7616d344e0ea893b46227440ef1/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/059ace63cca414c0b2cfdff9df65ba6bf009d06d374a5a43d5fc33edeb68c7d9/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1ca9416f4f24018 MacAddress:ee:06:0f:e8:e6:9e Speed:10000 Mtu:8901} {Name:26e2d32ed3f80f3 MacAddress:f2:ed:45:b3:e0:77 Speed:10000 Mtu:8901} {Name:6accded0538df02 MacAddress:f6:a7:1b:64:25:cc Speed:10000 Mtu:8901} {Name:7149a3ee7166a85 MacAddress:5e:b3:f2:b3:18:5b Speed:10000 Mtu:8901} {Name:75a18ed8c316584 MacAddress:42:58:59:08:ce:cc Speed:10000 Mtu:8901} {Name:7850e679b9b2db9 MacAddress:aa:85:e7:27:ef:dd Speed:10000 Mtu:8901} {Name:80dc414f095cf57 MacAddress:f2:8e:7b:60:49:f4 Speed:10000 Mtu:8901} {Name:893a64eb71fdd26 MacAddress:b6:69:4c:e8:a1:8a Speed:10000 Mtu:8901} {Name:9d07cd3031c2ed2 MacAddress:66:63:1e:fb:b9:83 Speed:10000 Mtu:8901} {Name:affcf81e798d485 MacAddress:82:04:56:ae:c8:83 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e6:f9:02:b3 Speed:0 Mtu:9001} {Name:br-int MacAddress:76:29:20:6c:d1:63 Speed:0 Mtu:8901} {Name:da3861e8abc8d28 MacAddress:a2:2a:07:7f:b1:d7 Speed:10000 Mtu:8901} {Name:e6088cca271321b MacAddress:d6:90:6a:43:cd:84 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e6:f9:02:b3 Speed:0 Mtu:9001} {Name:fc26bdf4cad8d17 MacAddress:2e:48:00:f5:34:09 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:36:cb:a4:09:2f:9a Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:9e:62:be:ee:9f:be Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 19:56:57.160083 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.157498 4918 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 19:56:57.160083 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.157542 4918 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 19:56:57.160083 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.157802 4918 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 19:56:57.160083 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.157822 4918 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-53.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 19:56:57.160083 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.157967 4918 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 19:56:57.160083 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.157974 4918 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 19:56:57.160083 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.157987 4918 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:56:57.160083 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.158007 4918 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:56:57.160083 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.158099 4918 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 19:56:57.160083 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.158275 4918 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 19:56:57.160083 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.158379 4918 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 19:56:57.160083 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.158413 4918 kubelet.go:493] "Attempting to sync node with API server" Nov 03 19:56:57.160083 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.158426 4918 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 19:56:57.160083 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.158445 4918 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 19:56:57.160083 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.158454 4918 kubelet.go:405] "Adding apiserver pod source" Nov 03 19:56:57.160083 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.158462 4918 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 19:56:57.160083 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.159524 4918 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 19:56:57.160083 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.159538 4918 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:56:57.160890 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.160077 4918 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 19:56:57.163678 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.163660 4918 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 19:56:57.163678 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.163673 4918 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:56:57.163799 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.163764 4918 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 19:56:57.163954 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.163941 4918 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 19:56:57.163985 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.163971 4918 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 19:56:57.164065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.164057 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 19:56:57.164100 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.164070 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 19:56:57.164100 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.164077 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 19:56:57.164100 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.164082 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 19:56:57.164100 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.164088 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 19:56:57.164100 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.164095 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 19:56:57.164100 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.164100 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 19:56:57.164492 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.164106 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 19:56:57.164492 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.164113 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 19:56:57.164492 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.164118 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 19:56:57.164492 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.164127 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 19:56:57.164492 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.164136 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 19:56:57.164492 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.164160 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 19:56:57.164492 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.164166 4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 19:56:57.167586 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.167571 4918 server.go:1263] "Started kubelet" Nov 03 19:56:57.167726 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.167649 4918 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 19:56:57.167884 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.167792 4918 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 19:56:57.168162 ip-10-0-139-53 systemd[1]: Started Kubernetes Kubelet. Nov 03 19:56:57.168356 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.167862 4918 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 19:56:57.168691 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.168653 4918 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 19:56:57.170543 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.170528 4918 server.go:317] "Adding debug handlers to kubelet server" Nov 03 19:56:57.171660 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.171641 4918 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 19:56:57.172559 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.171658 4918 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 19:56:57.172559 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.171702 4918 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 19:56:57.172559 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.171710 4918 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 19:56:57.172559 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.171791 4918 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 19:56:57.172559 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.171700 4918 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:51:17 +0000 UTC" deadline="2028-08-05 05:15:37.899943536 +0000 UTC" Nov 03 19:56:57.172559 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.171818 4918 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15369h18m40.728130211s" Nov 03 19:56:57.172559 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:57.172109 4918 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-53.us-east-2.compute.internal\" not found" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176506 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adc4cee9-00f0-4fac-b655-f58c21377362" volumeName="kubernetes.io/secret/adc4cee9-00f0-4fac-b655-f58c21377362-image-registry-private-configuration" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176578 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aed0ee7d-d648-4138-b5ef-2ae6795321d2" volumeName="kubernetes.io/projected/aed0ee7d-d648-4138-b5ef-2ae6795321d2-kube-api-access-w5sh5" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176599 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f838cffc-2fc7-4a3f-880b-8a80298cb1b4" volumeName="kubernetes.io/empty-dir/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-tmp" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176614 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433c5b6b-176b-44d1-afc0-6884045e468c" volumeName="kubernetes.io/secret/433c5b6b-176b-44d1-afc0-6884045e468c-serving-cert" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176627 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4726eb4c-48e8-423d-a62c-250a18bd8b83" volumeName="kubernetes.io/projected/4726eb4c-48e8-423d-a62c-250a18bd8b83-kube-api-access-ml5nd" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176658 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2e4493e-d29f-49b6-851e-cbd01214bf5c" volumeName="kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176672 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="158d8c01-2d31-43b4-a098-0c731c1fd163" volumeName="kubernetes.io/projected/158d8c01-2d31-43b4-a098-0c731c1fd163-kube-api-access-wl89h" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176689 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="449a05ee-ead2-4507-8d65-faaeb3ab3281" volumeName="kubernetes.io/empty-dir/449a05ee-ead2-4507-8d65-faaeb3ab3281-tmp" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176709 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99713fae-ae74-4da5-b435-c73edc1c4843" volumeName="kubernetes.io/configmap/99713fae-ae74-4da5-b435-c73edc1c4843-env-overrides" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176729 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99713fae-ae74-4da5-b435-c73edc1c4843" volumeName="kubernetes.io/secret/99713fae-ae74-4da5-b435-c73edc1c4843-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176743 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff051e4a-95a6-4c0d-9f69-773974d6eb7c" volumeName="kubernetes.io/configmap/ff051e4a-95a6-4c0d-9f69-773974d6eb7c-config" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176756 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff051e4a-95a6-4c0d-9f69-773974d6eb7c" volumeName="kubernetes.io/secret/ff051e4a-95a6-4c0d-9f69-773974d6eb7c-serving-cert" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176776 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="653cb377-919b-4793-85a4-0d60d6aeaff6" volumeName="kubernetes.io/projected/653cb377-919b-4793-85a4-0d60d6aeaff6-kube-api-access-vb6j8" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176805 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88c8909b-9acd-46d0-9f19-d74e2db51f77" volumeName="kubernetes.io/configmap/88c8909b-9acd-46d0-9f19-d74e2db51f77-cni-binary-copy" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176829 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99713fae-ae74-4da5-b435-c73edc1c4843" volumeName="kubernetes.io/projected/99713fae-ae74-4da5-b435-c73edc1c4843-kube-api-access-ctj28" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176845 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f90a0a4e-2794-46fc-a799-9c1593d859ab" volumeName="kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176864 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33b6188c-0202-4bc8-813f-2d189587f3aa" volumeName="kubernetes.io/empty-dir/33b6188c-0202-4bc8-813f-2d189587f3aa-archives-path" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176879 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="514e4bfb-77ef-4b1f-977a-3f9308cf886c" volumeName="kubernetes.io/configmap/514e4bfb-77ef-4b1f-977a-3f9308cf886c-trusted-ca" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176894 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="653cb377-919b-4793-85a4-0d60d6aeaff6" volumeName="kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-stats-auth" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176912 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6de4fd5d-d10a-4630-80bf-3838fb02d679" volumeName="kubernetes.io/projected/6de4fd5d-d10a-4630-80bf-3838fb02d679-kube-api-access-8f594" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176926 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93a3cf28-8c0c-4735-b177-d28bc7c8ea3f" volumeName="kubernetes.io/projected/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-kube-api-access-dd49x" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176945 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab34f1da-ace1-443d-bddd-9d9905d22e52" volumeName="kubernetes.io/projected/ab34f1da-ace1-443d-bddd-9d9905d22e52-kube-api-access-brzfd" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176957 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adc4cee9-00f0-4fac-b655-f58c21377362" volumeName="kubernetes.io/configmap/adc4cee9-00f0-4fac-b655-f58c21377362-trusted-ca" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176969 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22" volumeName="kubernetes.io/configmap/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-nginx-conf" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.176987 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fc8cec5-9248-4a47-822e-d81b37a380ef" volumeName="kubernetes.io/projected/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-api-access-kzdnd" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177000 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="449a05ee-ead2-4507-8d65-faaeb3ab3281" volumeName="kubernetes.io/projected/449a05ee-ead2-4507-8d65-faaeb3ab3281-kube-api-access-zxkg4" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177019 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="449a05ee-ead2-4507-8d65-faaeb3ab3281" volumeName="kubernetes.io/secret/449a05ee-ead2-4507-8d65-faaeb3ab3281-serving-cert" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177034 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="514e4bfb-77ef-4b1f-977a-3f9308cf886c" volumeName="kubernetes.io/secret/514e4bfb-77ef-4b1f-977a-3f9308cf886c-installation-pull-secrets" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177060 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adc4cee9-00f0-4fac-b655-f58c21377362" volumeName="kubernetes.io/configmap/adc4cee9-00f0-4fac-b655-f58c21377362-registry-certificates" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177088 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adc4cee9-00f0-4fac-b655-f58c21377362" volumeName="kubernetes.io/empty-dir/adc4cee9-00f0-4fac-b655-f58c21377362-ca-trust-extracted" seLinuxMountContext="" Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177105 4918 factory.go:55] Registering systemd factory Nov 03 19:56:57.177132 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177127 4918 factory.go:223] Registration of the systemd container factory successfully Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177105 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7dde7dd-245b-4437-9464-0e8b739caa98" volumeName="kubernetes.io/configmap/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-daemon-config" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177227 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26ac4e9f-3445-4994-aa1b-086b39b8c1cc" volumeName="kubernetes.io/configmap/26ac4e9f-3445-4994-aa1b-086b39b8c1cc-serviceca" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177248 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33b6188c-0202-4bc8-813f-2d189587f3aa" volumeName="kubernetes.io/configmap/33b6188c-0202-4bc8-813f-2d189587f3aa-service-ca-bundle" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177266 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433c5b6b-176b-44d1-afc0-6884045e468c" volumeName="kubernetes.io/projected/433c5b6b-176b-44d1-afc0-6884045e468c-kube-api-access-8x4lc" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177279 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="514e4bfb-77ef-4b1f-977a-3f9308cf886c" volumeName="kubernetes.io/configmap/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-certificates" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177292 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40" volumeName="kubernetes.io/projected/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40-kube-api-access-m7vhv" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177309 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88c8909b-9acd-46d0-9f19-d74e2db51f77" volumeName="kubernetes.io/configmap/88c8909b-9acd-46d0-9f19-d74e2db51f77-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177321 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88c8909b-9acd-46d0-9f19-d74e2db51f77" volumeName="kubernetes.io/configmap/88c8909b-9acd-46d0-9f19-d74e2db51f77-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177338 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93a3cf28-8c0c-4735-b177-d28bc7c8ea3f" volumeName="kubernetes.io/configmap/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-config-volume" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177351 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23f64ae9-fdda-4f2e-b3a5-1c552cc55a44" volumeName="kubernetes.io/configmap/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44-config" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177363 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="653cb377-919b-4793-85a4-0d60d6aeaff6" volumeName="kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-default-certificate" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177380 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40" volumeName="kubernetes.io/empty-dir/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40-tmp-dir" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177392 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a24da6cd-d805-4f8a-ac99-89dbd45c86c3" volumeName="kubernetes.io/configmap/a24da6cd-d805-4f8a-ac99-89dbd45c86c3-iptables-alerter-script" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177410 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93a3cf28-8c0c-4735-b177-d28bc7c8ea3f" volumeName="kubernetes.io/empty-dir/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-tmp-dir" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177424 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcfef4da-a4d7-4ef0-a383-ff2acb8cac79" volumeName="kubernetes.io/configmap/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-config" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177436 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d9796ba-d26b-494e-a4b1-2ebab6496f49" volumeName="kubernetes.io/projected/0d9796ba-d26b-494e-a4b1-2ebab6496f49-kube-api-access-pgswz" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177453 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23f64ae9-fdda-4f2e-b3a5-1c552cc55a44" volumeName="kubernetes.io/secret/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44-serving-cert" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177465 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff051e4a-95a6-4c0d-9f69-773974d6eb7c" volumeName="kubernetes.io/projected/ff051e4a-95a6-4c0d-9f69-773974d6eb7c-kube-api-access-gm2tl" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177482 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33b6188c-0202-4bc8-813f-2d189587f3aa" volumeName="kubernetes.io/configmap/33b6188c-0202-4bc8-813f-2d189587f3aa-trusted-ca-bundle" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177500 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61f1d7a9-a8b3-4db5-878d-4b8d1632fafc" volumeName="kubernetes.io/projected/61f1d7a9-a8b3-4db5-878d-4b8d1632fafc-kube-api-access-jh8xj" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177512 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a24da6cd-d805-4f8a-ac99-89dbd45c86c3" volumeName="kubernetes.io/projected/a24da6cd-d805-4f8a-ac99-89dbd45c86c3-kube-api-access-5jhkj" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177529 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adc4cee9-00f0-4fac-b655-f58c21377362" volumeName="kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-bound-sa-token" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177542 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7dde7dd-245b-4437-9464-0e8b739caa98" volumeName="kubernetes.io/configmap/f7dde7dd-245b-4437-9464-0e8b739caa98-cni-binary-copy" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177558 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcfef4da-a4d7-4ef0-a383-ff2acb8cac79" volumeName="kubernetes.io/projected/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-kube-api-access-gtdbf" seLinuxMountContext="" Nov 03 19:56:57.178565 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177572 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23f64ae9-fdda-4f2e-b3a5-1c552cc55a44" volumeName="kubernetes.io/projected/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44-kube-api-access-6hb8f" seLinuxMountContext="" Nov 03 19:56:57.179622 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.178676 4918 factory.go:153] Registering CRI-O factory Nov 03 19:56:57.179622 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.178700 4918 factory.go:223] Registration of the crio container factory successfully Nov 03 19:56:57.179622 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.178775 4918 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 19:56:57.179622 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.178795 4918 factory.go:103] Registering Raw factory Nov 03 19:56:57.179622 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.178819 4918 manager.go:1199] Started watching for new ooms in manager Nov 03 19:56:57.179622 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.179203 4918 manager.go:319] Starting recovery of all containers Nov 03 19:56:57.179622 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.177585 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433c5b6b-176b-44d1-afc0-6884045e468c" volumeName="kubernetes.io/configmap/433c5b6b-176b-44d1-afc0-6884045e468c-config" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185296 4918 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185374 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="514e4bfb-77ef-4b1f-977a-3f9308cf886c" volumeName="kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-kube-api-access-djg6c" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185405 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88c8909b-9acd-46d0-9f19-d74e2db51f77" volumeName="kubernetes.io/projected/88c8909b-9acd-46d0-9f19-d74e2db51f77-kube-api-access-7g8dn" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185420 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96087edc-f246-4ba4-a9d6-734a997da1f9" volumeName="kubernetes.io/secret/96087edc-f246-4ba4-a9d6-734a997da1f9-agent-certs" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185434 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7dde7dd-245b-4437-9464-0e8b739caa98" volumeName="kubernetes.io/projected/f7dde7dd-245b-4437-9464-0e8b739caa98-kube-api-access-z6ggm" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185447 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f838cffc-2fc7-4a3f-880b-8a80298cb1b4" volumeName="kubernetes.io/projected/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-kube-api-access-sk6q8" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185458 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="449a05ee-ead2-4507-8d65-faaeb3ab3281" volumeName="kubernetes.io/empty-dir/449a05ee-ead2-4507-8d65-faaeb3ab3281-snapshots" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185471 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adc4cee9-00f0-4fac-b655-f58c21377362" volumeName="kubernetes.io/secret/adc4cee9-00f0-4fac-b655-f58c21377362-installation-pull-secrets" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185484 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fc8cec5-9248-4a47-822e-d81b37a380ef" volumeName="kubernetes.io/empty-dir/1fc8cec5-9248-4a47-822e-d81b37a380ef-data-volume" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185495 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26ac4e9f-3445-4994-aa1b-086b39b8c1cc" volumeName="kubernetes.io/projected/26ac4e9f-3445-4994-aa1b-086b39b8c1cc-kube-api-access-7bqcz" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185510 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99713fae-ae74-4da5-b435-c73edc1c4843" volumeName="kubernetes.io/configmap/99713fae-ae74-4da5-b435-c73edc1c4843-ovnkube-config" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185524 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c60cb950-5dd9-4a2b-bba3-b3c38d3591e3" volumeName="kubernetes.io/projected/c60cb950-5dd9-4a2b-bba3-b3c38d3591e3-kube-api-access-6vgq9" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185538 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="449a05ee-ead2-4507-8d65-faaeb3ab3281" volumeName="kubernetes.io/configmap/449a05ee-ead2-4507-8d65-faaeb3ab3281-trusted-ca-bundle" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185551 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="514e4bfb-77ef-4b1f-977a-3f9308cf886c" volumeName="kubernetes.io/empty-dir/514e4bfb-77ef-4b1f-977a-3f9308cf886c-ca-trust-extracted" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185566 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a7d187d-9a17-475b-8ee4-ae5836d9a73a" volumeName="kubernetes.io/projected/5a7d187d-9a17-475b-8ee4-ae5836d9a73a-kube-api-access-gkf49" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185581 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adc4cee9-00f0-4fac-b655-f58c21377362" volumeName="kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-kube-api-access-6sx9r" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185596 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aed0ee7d-d648-4138-b5ef-2ae6795321d2" volumeName="kubernetes.io/secret/aed0ee7d-d648-4138-b5ef-2ae6795321d2-cert" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185610 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f838cffc-2fc7-4a3f-880b-8a80298cb1b4" volumeName="kubernetes.io/empty-dir/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-tuned" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185625 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433c5b6b-176b-44d1-afc0-6884045e468c" volumeName="kubernetes.io/configmap/433c5b6b-176b-44d1-afc0-6884045e468c-trusted-ca" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185639 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="449a05ee-ead2-4507-8d65-faaeb3ab3281" volumeName="kubernetes.io/configmap/449a05ee-ead2-4507-8d65-faaeb3ab3281-service-ca-bundle" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185655 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4726eb4c-48e8-423d-a62c-250a18bd8b83" volumeName="kubernetes.io/configmap/4726eb4c-48e8-423d-a62c-250a18bd8b83-telemetry-config" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185668 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="514e4bfb-77ef-4b1f-977a-3f9308cf886c" volumeName="kubernetes.io/secret/514e4bfb-77ef-4b1f-977a-3f9308cf886c-image-registry-private-configuration" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185684 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96087edc-f246-4ba4-a9d6-734a997da1f9" volumeName="kubernetes.io/configmap/96087edc-f246-4ba4-a9d6-734a997da1f9-konnectivity-ca" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185698 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99713fae-ae74-4da5-b435-c73edc1c4843" volumeName="kubernetes.io/configmap/99713fae-ae74-4da5-b435-c73edc1c4843-ovnkube-script-lib" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185712 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33b6188c-0202-4bc8-813f-2d189587f3aa" volumeName="kubernetes.io/projected/33b6188c-0202-4bc8-813f-2d189587f3aa-kube-api-access-gxblp" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185734 4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="514e4bfb-77ef-4b1f-977a-3f9308cf886c" volumeName="kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-bound-sa-token" seLinuxMountContext="" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185747 4918 reconstruct.go:97] "Volume reconstruction finished" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.185755 4918 reconciler.go:29] "Reconciler: start to sync state" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.186808 4918 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:57.187065 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.187007 4918 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:57.189101 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.189079 4918 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 19:56:57.196499 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.196483 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/1.log" Nov 03 19:56:57.196576 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.196523 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/1.log" Nov 03 19:56:57.204819 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.204782 4918 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 19:56:57.206492 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.206474 4918 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 19:56:57.206611 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.206589 4918 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 19:56:57.206724 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.206620 4918 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 19:56:57.206724 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:57.206659 4918 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 19:56:57.210376 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.210353 4918 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:57.212860 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.212838 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/1.log" Nov 03 19:56:57.212955 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.212868 4918 generic.go:356] "Generic (PLEG): container finished" podID="433c5b6b-176b-44d1-afc0-6884045e468c" containerID="33c82892c5f4b399f0d1f3ee43a5aee3b52b13b00868cd4bbdb8ac251bc91a2f" exitCode=255 Nov 03 19:56:57.248440 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.248413 4918 generic.go:356] "Generic (PLEG): container finished" podID="88c8909b-9acd-46d0-9f19-d74e2db51f77" containerID="396c87eeab78084d270a553e6d86c9263cdacbf1f6382b8183d7db97fa2bd9c3" exitCode=0 Nov 03 19:56:57.248440 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.248439 4918 generic.go:356] "Generic (PLEG): container finished" podID="88c8909b-9acd-46d0-9f19-d74e2db51f77" containerID="10a92b2b943ae59e4711069fc7cd1fe05a72f072578820e08714063a9bcc0636" exitCode=0 Nov 03 19:56:57.248619 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.248450 4918 generic.go:356] "Generic (PLEG): container finished" podID="88c8909b-9acd-46d0-9f19-d74e2db51f77" containerID="a38de7e690d459b8b35791ed408515ab1910b5cfd4405a986c8f2c5a04ab93f7" exitCode=0 Nov 03 19:56:57.248619 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.248458 4918 generic.go:356] "Generic (PLEG): container finished" podID="88c8909b-9acd-46d0-9f19-d74e2db51f77" containerID="08632f5eea60d7461fb62d450fbc7f7b62ecc39ec0e958b9fcc14d8c9ce16398" exitCode=0 Nov 03 19:56:57.248619 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.248467 4918 generic.go:356] "Generic (PLEG): container finished" podID="88c8909b-9acd-46d0-9f19-d74e2db51f77" containerID="a1905c04f9a2c1037bdea9d29299cd8cb1ba04936eb8e822d67b5d1332c6f26e" exitCode=0 Nov 03 19:56:57.248619 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.248472 4918 generic.go:356] "Generic (PLEG): container finished" podID="88c8909b-9acd-46d0-9f19-d74e2db51f77" containerID="c7d5d96db7a890be66ed19320e9d376743ea9bcc10db797e33b5cfc7940130a7" exitCode=0 Nov 03 19:56:57.250697 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.250627 4918 generic.go:356] "Generic (PLEG): container finished" podID="13c920c8ddfef32d3fa81fbcf93279d1" containerID="3c09f2ddd018c8267a595104d4a4d2ca84b58de70c2aa17ac9750bc1cb1e1c1c" exitCode=0 Nov 03 19:56:57.274719 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.274699 4918 manager.go:324] Recovery completed Nov 03 19:56:57.295155 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.295126 4918 cpu_manager.go:226] "Starting" policy="none" Nov 03 19:56:57.295155 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.295147 4918 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 19:56:57.295330 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.295167 4918 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 19:56:57.295330 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.295322 4918 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 19:56:57.295406 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.295334 4918 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 19:56:57.295406 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.295398 4918 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 19:56:57.295406 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.295404 4918 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 19:56:57.295486 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.295409 4918 policy_none.go:50] "Start" Nov 03 19:56:57.295486 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.295418 4918 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 19:56:57.295486 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.295427 4918 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 19:56:57.295570 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.295546 4918 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 19:56:57.295570 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.295553 4918 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 19:56:57.295570 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.295558 4918 policy_none.go:44] "Start" Nov 03 19:56:57.299869 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.299840 4918 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 19:56:57.302959 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.302943 4918 manager.go:341] "Starting Device Plugin manager" Nov 03 19:56:57.303669 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:57.302978 4918 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 19:56:57.303707 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.303677 4918 server.go:87] "Starting device plugin registration server" Nov 03 19:56:57.304023 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.304010 4918 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 19:56:57.304061 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.304027 4918 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 19:56:57.304108 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.304092 4918 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 19:56:57.304226 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.304190 4918 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 19:56:57.304328 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.304237 4918 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 19:56:57.304328 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.304244 4918 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 19:56:57.304422 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.304294 4918 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:56:57.304220134Z","UUID":"5de77402-8f42-4e8a-9d2f-33b114c23eb0","Handler":null,"Name":"","Endpoint":""} Nov 03 19:56:57.305529 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.305511 4918 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 19:56:57.305635 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.305538 4918 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 19:56:57.305698 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:57.305680 4918 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 19:56:57.306873 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.306834 4918 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal"] Nov 03 19:56:57.308118 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.308096 4918 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:57.308350 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.308304 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal" event={"ID":"cb28595e81de1584c6e009472415b9b8","Type":"ContainerStarted","Data":"fab49129cb63d95837398d02a8a0e4411b864b0e1e4b6c630e326b2582d5b903"} Nov 03 19:56:57.308419 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.308370 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal" event={"ID":"cb28595e81de1584c6e009472415b9b8","Type":"ContainerStarted","Data":"5440d31e29ee75e3c31b392353d75842f76a00ab037294e19e4faa80f7f22d33"} Nov 03 19:56:57.308513 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.308486 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" event={"ID":"13c920c8ddfef32d3fa81fbcf93279d1","Type":"ContainerStarted","Data":"1638aced06b221e3add91dba0bd53e486df3bd54113e51010e31606bce114efa"} Nov 03 19:56:57.308627 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.308516 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" event={"ID":"13c920c8ddfef32d3fa81fbcf93279d1","Type":"ContainerDied","Data":"3c09f2ddd018c8267a595104d4a4d2ca84b58de70c2aa17ac9750bc1cb1e1c1c"} Nov 03 19:56:57.308627 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.308541 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" event={"ID":"13c920c8ddfef32d3fa81fbcf93279d1","Type":"ContainerStarted","Data":"676071769d6a0e543d6500b74fd35bf48763179597301db835c32ceb21c72e95"} Nov 03 19:56:57.312207 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.308976 4918 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:57.323507 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.323484 4918 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:56:57.323582 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:57.323540 4918 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:57.324057 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.324039 4918 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:56:57.324150 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:57.324078 4918 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:57.390271 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.390230 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13c920c8ddfef32d3fa81fbcf93279d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal\" (UID: \"13c920c8ddfef32d3fa81fbcf93279d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:57.390424 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.390275 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13c920c8ddfef32d3fa81fbcf93279d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal\" (UID: \"13c920c8ddfef32d3fa81fbcf93279d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:57.390424 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.390302 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cb28595e81de1584c6e009472415b9b8-config\") pod \"kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal\" (UID: \"cb28595e81de1584c6e009472415b9b8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:57.404315 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.404291 4918 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:57.405675 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.405651 4918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:57.405759 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.405686 4918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:57.405759 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.405698 4918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-53.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:57.405759 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.405738 4918 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:57.418130 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.418105 4918 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:57.418259 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.418171 4918 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:57.491453 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.491420 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cb28595e81de1584c6e009472415b9b8-config\") pod \"kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal\" (UID: \"cb28595e81de1584c6e009472415b9b8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:57.491596 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.491454 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13c920c8ddfef32d3fa81fbcf93279d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal\" (UID: \"13c920c8ddfef32d3fa81fbcf93279d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:57.491596 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.491490 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13c920c8ddfef32d3fa81fbcf93279d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal\" (UID: \"13c920c8ddfef32d3fa81fbcf93279d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:57.491596 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.491494 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cb28595e81de1584c6e009472415b9b8-config\") pod \"kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal\" (UID: \"cb28595e81de1584c6e009472415b9b8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:57.491596 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.491531 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13c920c8ddfef32d3fa81fbcf93279d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal\" (UID: \"13c920c8ddfef32d3fa81fbcf93279d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:57.491596 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:57.491536 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13c920c8ddfef32d3fa81fbcf93279d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal\" (UID: \"13c920c8ddfef32d3fa81fbcf93279d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:58.159384 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.159343 4918 apiserver.go:52] "Watching apiserver" Nov 03 19:56:58.172356 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.172328 4918 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 19:56:58.173051 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.173027 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6r7g9","openshift-insights/insights-operator-7c4986d449-zl889","openshift-insights/insights-runtime-extractor-k9x2f","openshift-ingress-canary/ingress-canary-95k62","kube-system/kas-connection-checker-685895c7c9-qkzrh","openshift-network-console/networking-console-plugin-86c478bb88-2zqfd","openshift-ovn-kubernetes/ovnkube-node-bvlzl","openshift-multus/multus-9gbmz","kube-system/kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal","openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st","openshift-network-operator/iptables-alerter-fjr5q","openshift-multus/network-metrics-daemon-dcqpv","openshift-image-registry/image-registry-9d99c5c47-ttssq","kube-system/kas-connection-checker-685895c7c9-5dgdg","openshift-network-diagnostics/network-check-target-zn2ms","kube-system/konnectivity-agent-ntz8w","openshift-cluster-node-tuning-operator/tuned-5vpvg","openshift-dns/dns-default-xsh7l","openshift-dns/node-resolver-ksm7q","openshift-image-registry/node-ca-jjkqt","openshift-network-diagnostics/network-check-source-6f78d59f99-qtbqg","kube-system/global-pull-secret-syncer-mvsdz","openshift-insights/periodic-gathering-l984r-mntts","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mvxwm","openshift-console-operator/console-operator-7cd56b8dbc-jk8h6","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq","openshift-multus/multus-additional-cni-plugins-wjgg6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal","openshift-ingress/router-default-6bfcd89b85-9qb4s","openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk","openshift-image-registry/image-registry-5d95b6bf4b-dd7kz"] Nov 03 19:56:58.175499 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.175473 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:58.177426 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.177400 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:58.177521 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.177424 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 19:56:58.177645 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.177628 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 19:56:58.178389 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.178366 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:56:58.178889 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.178864 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 19:56:58.179031 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.179015 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 19:56:58.179243 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.179228 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:56:58.179533 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.179514 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 19:56:58.179625 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.179593 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 19:56:58.179706 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.179686 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 19:56:58.179770 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.179747 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nshbb\"" Nov 03 19:56:58.179886 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.179869 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qbp66\"" Nov 03 19:56:58.180251 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.180232 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:56:58.180367 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.179908 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:58.180519 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.180469 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 19:56:58.180519 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.180503 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 19:56:58.180689 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.180577 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jwsg8\"" Nov 03 19:56:58.180819 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.180801 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 19:56:58.181004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.180969 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9lnsw\"" Nov 03 19:56:58.181167 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.181151 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:56:58.181296 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.181279 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 19:56:58.181340 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.181305 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-f6mkl\"" Nov 03 19:56:58.181754 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.181738 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 19:56:58.182126 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.182107 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 19:56:58.182382 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.182251 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:56:58.182382 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.182366 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 19:56:58.182539 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.182514 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 19:56:58.182616 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.182551 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 19:56:58.182742 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.182730 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:58.183529 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.183513 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 19:56:58.183633 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.183592 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ft7x5\"" Nov 03 19:56:58.183866 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.183800 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 19:56:58.183989 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.183937 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 19:56:58.184096 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.184053 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 19:56:58.184096 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.184080 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dfmhm\"" Nov 03 19:56:58.184446 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.184416 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 19:56:58.184446 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.184431 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 19:56:58.184446 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.184434 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 19:56:58.184446 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.184451 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 19:56:58.185980 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.185965 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:58.186957 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.186940 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 19:56:58.189274 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.189244 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:58.190448 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.190429 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:58.191037 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.191015 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:56:58.191118 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.191080 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 19:56:58.191338 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.191321 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 19:56:58.191491 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.191473 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5rc2l\"" Nov 03 19:56:58.191989 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.191975 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 19:56:58.192043 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.192027 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 19:56:58.192244 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.192228 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 19:56:58.192641 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.192623 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 19:56:58.194228 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.194208 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:58.194315 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.194300 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 19:56:58.196984 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.196962 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 19:56:58.197098 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.197082 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 19:56:58.197410 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.197386 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 19:56:58.197500 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.197413 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 19:56:58.197500 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.197473 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 19:56:58.197594 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.197538 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 19:56:58.197594 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.197562 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:56:58.197594 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.197577 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 19:56:58.197713 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.197615 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 19:56:58.197713 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.197622 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 19:56:58.197713 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.197667 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 19:56:58.197713 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.197677 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 19:56:58.197873 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.197723 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 19:56:58.197873 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.197766 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 19:56:58.197873 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.197847 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 19:56:58.197996 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.197936 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 19:56:58.197996 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.197980 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 19:56:58.198079 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.197997 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 19:56:58.198079 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.198025 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 19:56:58.198079 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.198036 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:56:58.198224 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.198161 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 19:56:58.198224 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.198206 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 19:56:58.198363 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.198338 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 19:56:58.198500 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.198477 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 19:56:58.198577 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.198509 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-72cwb\"" Nov 03 19:56:58.198733 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.198712 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 19:56:58.198851 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.198770 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p44lh\"" Nov 03 19:56:58.198913 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.198772 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 19:56:58.198972 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.198918 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lp2l4\"" Nov 03 19:56:58.198972 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.198919 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sztcx\"" Nov 03 19:56:58.199075 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.199016 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 19:56:58.199154 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.199135 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 19:56:58.199244 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.199222 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 19:56:58.199244 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.199239 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6tnlf\"" Nov 03 19:56:58.199351 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.199145 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n8st4\"" Nov 03 19:56:58.199351 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.199148 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gc46m\"" Nov 03 19:56:58.199351 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.199137 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hc4hl\"" Nov 03 19:56:58.199351 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.199146 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 19:56:58.199351 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.199316 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 19:56:58.199663 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.199646 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-xd6ln\"" Nov 03 19:56:58.199663 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.199658 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wsxsf\"" Nov 03 19:56:58.199762 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.199710 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 19:56:58.199762 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.199725 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4n975\"" Nov 03 19:56:58.199828 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.199819 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8rzcq\"" Nov 03 19:56:58.199904 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.199893 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6mlp2\"" Nov 03 19:56:58.200060 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.200049 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 19:56:58.200093 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.200069 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 19:56:58.200389 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.200374 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5vrk4\"" Nov 03 19:56:58.200460 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.200387 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l9cjs\"" Nov 03 19:56:58.200460 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.200445 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 19:56:58.202090 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.202077 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 19:56:58.202153 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.202135 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 19:56:58.202212 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.202148 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 19:56:58.202212 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.202204 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 19:56:58.202306 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.202294 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 19:56:58.202351 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.202298 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rpk57\"" Nov 03 19:56:58.202351 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.202337 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4mwrd\"" Nov 03 19:56:58.203543 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.203525 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 19:56:58.203609 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.203556 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 19:56:58.203609 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.203597 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 19:56:58.203708 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.203670 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kfdzx\"" Nov 03 19:56:58.260663 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.260641 4918 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:58.271981 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.271954 4918 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:56:58.272067 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.272005 4918 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.us-east-2.compute.internal" Nov 03 19:56:58.273657 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.273642 4918 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 19:56:58.295476 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.295451 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl89h\" (UniqueName: \"kubernetes.io/projected/158d8c01-2d31-43b4-a098-0c731c1fd163-kube-api-access-wl89h\") pod \"kas-connection-checker-685895c7c9-6r7g9\" (UID: \"158d8c01-2d31-43b4-a098-0c731c1fd163\") " pod="kube-system/kas-connection-checker-685895c7c9-6r7g9" Nov 03 19:56:58.295566 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.295485 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-slash\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.295566 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.295502 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96087edc-f246-4ba4-a9d6-734a997da1f9-agent-certs\") pod \"konnectivity-agent-ntz8w\" (UID: \"96087edc-f246-4ba4-a9d6-734a997da1f9\") " pod="kube-system/konnectivity-agent-ntz8w" Nov 03 19:56:58.295566 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.295519 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-hostroot\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.295566 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.295536 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/433c5b6b-176b-44d1-afc0-6884045e468c-config\") pod \"console-operator-7cd56b8dbc-jk8h6\" (UID: \"433c5b6b-176b-44d1-afc0-6884045e468c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:58.295703 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.295609 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:58.295703 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.295629 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-sysctl-d\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.295703 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.295664 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88c8909b-9acd-46d0-9f19-d74e2db51f77-system-cni-dir\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:58.295703 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.295682 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jhkj\" (UniqueName: \"kubernetes.io/projected/a24da6cd-d805-4f8a-ac99-89dbd45c86c3-kube-api-access-5jhkj\") pod \"iptables-alerter-fjr5q\" (UID: \"a24da6cd-d805-4f8a-ac99-89dbd45c86c3\") " pod="openshift-network-operator/iptables-alerter-fjr5q" Nov 03 19:56:58.295961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.295943 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/433c5b6b-176b-44d1-afc0-6884045e468c-config\") pod \"console-operator-7cd56b8dbc-jk8h6\" (UID: \"433c5b6b-176b-44d1-afc0-6884045e468c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:58.295998 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.295955 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-cnibin\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.295998 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.295991 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-stats-auth\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:58.296054 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.295998 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96087edc-f246-4ba4-a9d6-734a997da1f9-agent-certs\") pod \"konnectivity-agent-ntz8w\" (UID: \"96087edc-f246-4ba4-a9d6-734a997da1f9\") " pod="kube-system/konnectivity-agent-ntz8w" Nov 03 19:56:58.296054 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296013 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bqcz\" (UniqueName: \"kubernetes.io/projected/26ac4e9f-3445-4994-aa1b-086b39b8c1cc-kube-api-access-7bqcz\") pod \"node-ca-jjkqt\" (UID: \"26ac4e9f-3445-4994-aa1b-086b39b8c1cc\") " pod="openshift-image-registry/node-ca-jjkqt" Nov 03 19:56:58.296054 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296033 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88c8909b-9acd-46d0-9f19-d74e2db51f77-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:58.296136 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296057 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.296136 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296074 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:58.296235 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296139 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-run-netns\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.296235 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296202 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/99713fae-ae74-4da5-b435-c73edc1c4843-ovnkube-script-lib\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.296324 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296232 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-stats-auth\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:58.296324 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296250 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh8xj\" (UniqueName: \"kubernetes.io/projected/61f1d7a9-a8b3-4db5-878d-4b8d1632fafc-kube-api-access-jh8xj\") pod \"kas-connection-checker-685895c7c9-5dgdg\" (UID: \"61f1d7a9-a8b3-4db5-878d-4b8d1632fafc\") " pod="kube-system/kas-connection-checker-685895c7c9-5dgdg" Nov 03 19:56:58.296419 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296375 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33b6188c-0202-4bc8-813f-2d189587f3aa-trusted-ca-bundle\") pod \"periodic-gathering-l984r-mntts\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:58.296474 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296445 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-default-certificate\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:58.296474 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296464 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88c8909b-9acd-46d0-9f19-d74e2db51f77-cni-binary-copy\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:58.296573 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296485 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-tuned\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.296573 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296501 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/adc4cee9-00f0-4fac-b655-f58c21377362-registry-certificates\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:58.296573 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296530 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgswz\" (UniqueName: \"kubernetes.io/projected/0d9796ba-d26b-494e-a4b1-2ebab6496f49-kube-api-access-pgswz\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:58.296573 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296562 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g8dn\" (UniqueName: \"kubernetes.io/projected/88c8909b-9acd-46d0-9f19-d74e2db51f77-kube-api-access-7g8dn\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:58.296756 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296593 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/449a05ee-ead2-4507-8d65-faaeb3ab3281-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:58.296756 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296602 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-tuned\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.296756 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296620 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/99713fae-ae74-4da5-b435-c73edc1c4843-ovnkube-script-lib\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.296756 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296699 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33b6188c-0202-4bc8-813f-2d189587f3aa-trusted-ca-bundle\") pod \"periodic-gathering-l984r-mntts\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:58.296756 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296622 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6ggm\" (UniqueName: \"kubernetes.io/projected/f7dde7dd-245b-4437-9464-0e8b739caa98-kube-api-access-z6ggm\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.296756 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296729 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/adc4cee9-00f0-4fac-b655-f58c21377362-registry-certificates\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:58.297012 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296756 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88c8909b-9acd-46d0-9f19-d74e2db51f77-cnibin\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:58.297012 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296785 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d2e4493e-d29f-49b6-851e-cbd01214bf5c-dbus\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:58.297012 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296811 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-sys-fs\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:58.297012 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296816 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88c8909b-9acd-46d0-9f19-d74e2db51f77-cni-binary-copy\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:58.297012 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296841 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40-tmp-dir\") pod \"node-resolver-ksm7q\" (UID: \"81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40\") " pod="openshift-dns/node-resolver-ksm7q" Nov 03 19:56:58.297012 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296852 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-default-certificate\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:58.297012 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296875 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-run-k8s-cni-cncf-io\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.297012 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296913 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40-tmp-dir\") pod \"node-resolver-ksm7q\" (UID: \"81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40\") " pod="openshift-dns/node-resolver-ksm7q" Nov 03 19:56:58.297012 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296952 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aed0ee7d-d648-4138-b5ef-2ae6795321d2-cert\") pod \"ingress-canary-95k62\" (UID: \"aed0ee7d-d648-4138-b5ef-2ae6795321d2\") " pod="openshift-ingress-canary/ingress-canary-95k62" Nov 03 19:56:58.297012 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296999 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml5nd\" (UniqueName: \"kubernetes.io/projected/4726eb4c-48e8-423d-a62c-250a18bd8b83-kube-api-access-ml5nd\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:58.297487 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.296993 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/449a05ee-ead2-4507-8d65-faaeb3ab3281-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:58.297487 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297054 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtdbf\" (UniqueName: \"kubernetes.io/projected/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-kube-api-access-gtdbf\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:58.297487 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297079 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-sysctl-conf\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.297487 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297097 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsb75\" (UniqueName: \"kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75\") pod \"network-check-target-zn2ms\" (UID: \"f90a0a4e-2794-46fc-a799-9c1593d859ab\") " pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:58.297487 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297121 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-run-ovn-kubernetes\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.297487 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297143 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/adc4cee9-00f0-4fac-b655-f58c21377362-image-registry-private-configuration\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:58.297487 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297161 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-bound-sa-token\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:58.297487 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297217 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:58.297487 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297234 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aed0ee7d-d648-4138-b5ef-2ae6795321d2-cert\") pod \"ingress-canary-95k62\" (UID: \"aed0ee7d-d648-4138-b5ef-2ae6795321d2\") " pod="openshift-ingress-canary/ingress-canary-95k62" Nov 03 19:56:58.297487 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297243 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-systemd-units\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.297487 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297298 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctj28\" (UniqueName: \"kubernetes.io/projected/99713fae-ae74-4da5-b435-c73edc1c4843-kube-api-access-ctj28\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.297487 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297325 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brzfd\" (UniqueName: \"kubernetes.io/projected/ab34f1da-ace1-443d-bddd-9d9905d22e52-kube-api-access-brzfd\") pod \"network-metrics-daemon-dcqpv\" (UID: \"ab34f1da-ace1-443d-bddd-9d9905d22e52\") " pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:58.297487 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297352 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-registration-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:58.297487 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297389 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40-hosts-file\") pod \"node-resolver-ksm7q\" (UID: \"81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40\") " pod="openshift-dns/node-resolver-ksm7q" Nov 03 19:56:58.297487 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297431 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/adc4cee9-00f0-4fac-b655-f58c21377362-image-registry-private-configuration\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:58.297487 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297441 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-socket-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:58.298172 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297490 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-kubernetes\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.298172 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297527 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:58.298172 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297592 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-run-ovn\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.298172 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297621 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/514e4bfb-77ef-4b1f-977a-3f9308cf886c-image-registry-private-configuration\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:58.298172 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297678 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:58.298172 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297730 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/514e4bfb-77ef-4b1f-977a-3f9308cf886c-ca-trust-extracted\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:58.298172 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297793 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/514e4bfb-77ef-4b1f-977a-3f9308cf886c-ca-trust-extracted\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:58.298172 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297810 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2zqfd\" (UID: \"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:56:58.298172 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297836 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/514e4bfb-77ef-4b1f-977a-3f9308cf886c-image-registry-private-configuration\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:58.298172 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297842 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/26ac4e9f-3445-4994-aa1b-086b39b8c1cc-serviceca\") pod \"node-ca-jjkqt\" (UID: \"26ac4e9f-3445-4994-aa1b-086b39b8c1cc\") " pod="openshift-image-registry/node-ca-jjkqt" Nov 03 19:56:58.298172 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297885 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-run-openvswitch\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.298172 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297927 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/adc4cee9-00f0-4fac-b655-f58c21377362-installation-pull-secrets\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:58.298172 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.297982 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96087edc-f246-4ba4-a9d6-734a997da1f9-konnectivity-ca\") pod \"konnectivity-agent-ntz8w\" (UID: \"96087edc-f246-4ba4-a9d6-734a997da1f9\") " pod="kube-system/konnectivity-agent-ntz8w" Nov 03 19:56:58.298172 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298053 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-etc-kubernetes\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.298172 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298079 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-bound-sa-token\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:58.298172 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298079 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/26ac4e9f-3445-4994-aa1b-086b39b8c1cc-serviceca\") pod \"node-ca-jjkqt\" (UID: \"26ac4e9f-3445-4994-aa1b-086b39b8c1cc\") " pod="openshift-image-registry/node-ca-jjkqt" Nov 03 19:56:58.298172 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298094 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd49x\" (UniqueName: \"kubernetes.io/projected/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-kube-api-access-dd49x\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:58.298172 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298113 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d2e4493e-d29f-49b6-851e-cbd01214bf5c-kubelet-config\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:58.298172 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298136 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/adc4cee9-00f0-4fac-b655-f58c21377362-installation-pull-secrets\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:58.298172 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298165 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96087edc-f246-4ba4-a9d6-734a997da1f9-konnectivity-ca\") pod \"konnectivity-agent-ntz8w\" (UID: \"96087edc-f246-4ba4-a9d6-734a997da1f9\") " pod="kube-system/konnectivity-agent-ntz8w" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298224 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-cni-bin\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298243 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hb8f\" (UniqueName: \"kubernetes.io/projected/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44-kube-api-access-6hb8f\") pod \"service-ca-operator-544d749c8c-9ncj4\" (UID: \"23f64ae9-fdda-4f2e-b3a5-1c552cc55a44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298257 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-conf-dir\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298275 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/33b6188c-0202-4bc8-813f-2d189587f3aa-archives-path\") pod \"periodic-gathering-l984r-mntts\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298302 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb6j8\" (UniqueName: \"kubernetes.io/projected/653cb377-919b-4793-85a4-0d60d6aeaff6-kube-api-access-vb6j8\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298338 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-host\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298372 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-tmp\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298382 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/33b6188c-0202-4bc8-813f-2d189587f3aa-archives-path\") pod \"periodic-gathering-l984r-mntts\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298410 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ff051e4a-95a6-4c0d-9f69-773974d6eb7c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq\" (UID: \"ff051e4a-95a6-4c0d-9f69-773974d6eb7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298433 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-tmp\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298436 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/449a05ee-ead2-4507-8d65-faaeb3ab3281-tmp\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298452 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1fc8cec5-9248-4a47-822e-d81b37a380ef-data-volume\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298481 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-node-log\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298506 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-log-socket\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298507 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/449a05ee-ead2-4507-8d65-faaeb3ab3281-tmp\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298538 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44-config\") pod \"service-ca-operator-544d749c8c-9ncj4\" (UID: \"23f64ae9-fdda-4f2e-b3a5-1c552cc55a44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298547 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1fc8cec5-9248-4a47-822e-d81b37a380ef-data-volume\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298584 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298611 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ff051e4a-95a6-4c0d-9f69-773974d6eb7c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq\" (UID: \"ff051e4a-95a6-4c0d-9f69-773974d6eb7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298647 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4726eb4c-48e8-423d-a62c-250a18bd8b83-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298669 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-etc-openvswitch\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298685 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-var-lib-cni-multus\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298718 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5sh5\" (UniqueName: \"kubernetes.io/projected/aed0ee7d-d648-4138-b5ef-2ae6795321d2-kube-api-access-w5sh5\") pod \"ingress-canary-95k62\" (UID: \"aed0ee7d-d648-4138-b5ef-2ae6795321d2\") " pod="openshift-ingress-canary/ingress-canary-95k62" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298738 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44-config\") pod \"service-ca-operator-544d749c8c-9ncj4\" (UID: \"23f64ae9-fdda-4f2e-b3a5-1c552cc55a44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" Nov 03 19:56:58.298849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298828 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-kubelet\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.299597 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298932 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4726eb4c-48e8-423d-a62c-250a18bd8b83-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:58.299597 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298963 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/449a05ee-ead2-4507-8d65-faaeb3ab3281-snapshots\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:58.299597 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.298999 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/514e4bfb-77ef-4b1f-977a-3f9308cf886c-installation-pull-secrets\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:58.299597 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299023 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxblp\" (UniqueName: \"kubernetes.io/projected/33b6188c-0202-4bc8-813f-2d189587f3aa-kube-api-access-gxblp\") pod \"periodic-gathering-l984r-mntts\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:58.299597 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299081 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:58.299597 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299082 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/449a05ee-ead2-4507-8d65-faaeb3ab3281-snapshots\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:58.299597 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299204 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26ac4e9f-3445-4994-aa1b-086b39b8c1cc-host\") pod \"node-ca-jjkqt\" (UID: \"26ac4e9f-3445-4994-aa1b-086b39b8c1cc\") " pod="openshift-image-registry/node-ca-jjkqt" Nov 03 19:56:58.299597 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299230 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/449a05ee-ead2-4507-8d65-faaeb3ab3281-serving-cert\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:58.299597 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299255 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/514e4bfb-77ef-4b1f-977a-3f9308cf886c-installation-pull-secrets\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:58.299597 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299262 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a24da6cd-d805-4f8a-ac99-89dbd45c86c3-host-slash\") pod \"iptables-alerter-fjr5q\" (UID: \"a24da6cd-d805-4f8a-ac99-89dbd45c86c3\") " pod="openshift-network-operator/iptables-alerter-fjr5q" Nov 03 19:56:58.299597 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299355 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djg6c\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-kube-api-access-djg6c\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:58.299597 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299374 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/449a05ee-ead2-4507-8d65-faaeb3ab3281-serving-cert\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:58.299597 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299465 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:58.299597 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299508 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/88c8909b-9acd-46d0-9f19-d74e2db51f77-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:58.299597 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299535 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm2tl\" (UniqueName: \"kubernetes.io/projected/ff051e4a-95a6-4c0d-9f69-773974d6eb7c-kube-api-access-gm2tl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq\" (UID: \"ff051e4a-95a6-4c0d-9f69-773974d6eb7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299607 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-modprobe-d\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299651 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk6q8\" (UniqueName: \"kubernetes.io/projected/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-kube-api-access-sk6q8\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299705 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/88c8909b-9acd-46d0-9f19-d74e2db51f77-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299708 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs\") pod \"network-metrics-daemon-dcqpv\" (UID: \"ab34f1da-ace1-443d-bddd-9d9905d22e52\") " pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299750 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a24da6cd-d805-4f8a-ac99-89dbd45c86c3-iptables-alerter-script\") pod \"iptables-alerter-fjr5q\" (UID: \"a24da6cd-d805-4f8a-ac99-89dbd45c86c3\") " pod="openshift-network-operator/iptables-alerter-fjr5q" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299791 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7vhv\" (UniqueName: \"kubernetes.io/projected/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40-kube-api-access-m7vhv\") pod \"node-resolver-ksm7q\" (UID: \"81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40\") " pod="openshift-dns/node-resolver-ksm7q" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299825 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33b6188c-0202-4bc8-813f-2d189587f3aa-service-ca-bundle\") pod \"periodic-gathering-l984r-mntts\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299852 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299876 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/99713fae-ae74-4da5-b435-c73edc1c4843-ovn-node-metrics-cert\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299908 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299927 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxkg4\" (UniqueName: \"kubernetes.io/projected/449a05ee-ead2-4507-8d65-faaeb3ab3281-kube-api-access-zxkg4\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299945 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-var-lib-cni-bin\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299961 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2zqfd\" (UID: \"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299973 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a24da6cd-d805-4f8a-ac99-89dbd45c86c3-iptables-alerter-script\") pod \"iptables-alerter-fjr5q\" (UID: \"a24da6cd-d805-4f8a-ac99-89dbd45c86c3\") " pod="openshift-network-operator/iptables-alerter-fjr5q" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.299985 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/88c8909b-9acd-46d0-9f19-d74e2db51f77-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300009 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33b6188c-0202-4bc8-813f-2d189587f3aa-service-ca-bundle\") pod \"periodic-gathering-l984r-mntts\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300036 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-systemd\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300036 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/99713fae-ae74-4da5-b435-c73edc1c4843-ovn-node-metrics-cert\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300055 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300084 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-var-lib-openvswitch\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300106 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-var-lib-kubelet\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.300116 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300124 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff051e4a-95a6-4c0d-9f69-773974d6eb7c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq\" (UID: \"ff051e4a-95a6-4c0d-9f69-773974d6eb7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300143 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-run-systemd\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300165 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-etc-selinux\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300215 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2zqfd\" (UID: \"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300215 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/88c8909b-9acd-46d0-9f19-d74e2db51f77-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300224 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-daemon-config\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300288 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-certificates\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300330 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x4lc\" (UniqueName: \"kubernetes.io/projected/433c5b6b-176b-44d1-afc0-6884045e468c-kube-api-access-8x4lc\") pod \"console-operator-7cd56b8dbc-jk8h6\" (UID: \"433c5b6b-176b-44d1-afc0-6884045e468c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300347 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff051e4a-95a6-4c0d-9f69-773974d6eb7c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq\" (UID: \"ff051e4a-95a6-4c0d-9f69-773974d6eb7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300384 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-run-netns\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300416 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-certificates\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300462 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-daemon-config\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300470 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-lib-modules\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300506 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/99713fae-ae74-4da5-b435-c73edc1c4843-env-overrides\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300552 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/adc4cee9-00f0-4fac-b655-f58c21377362-ca-trust-extracted\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300656 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adc4cee9-00f0-4fac-b655-f58c21377362-trusted-ca\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300667 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/99713fae-ae74-4da5-b435-c73edc1c4843-env-overrides\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300686 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/adc4cee9-00f0-4fac-b655-f58c21377362-ca-trust-extracted\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300725 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-cni-dir\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300821 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkf49\" (UniqueName: \"kubernetes.io/projected/5a7d187d-9a17-475b-8ee4-ae5836d9a73a-kube-api-access-gkf49\") pod \"kas-connection-checker-685895c7c9-qkzrh\" (UID: \"5a7d187d-9a17-475b-8ee4-ae5836d9a73a\") " pod="kube-system/kas-connection-checker-685895c7c9-qkzrh" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300841 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vgq9\" (UniqueName: \"kubernetes.io/projected/c60cb950-5dd9-4a2b-bba3-b3c38d3591e3-kube-api-access-6vgq9\") pod \"volume-data-source-validator-58c9f5f46d-mvxwm\" (UID: \"c60cb950-5dd9-4a2b-bba3-b3c38d3591e3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mvxwm" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300874 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/449a05ee-ead2-4507-8d65-faaeb3ab3281-service-ca-bundle\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300908 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1fc8cec5-9248-4a47-822e-d81b37a380ef-crio-socket\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300936 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adc4cee9-00f0-4fac-b655-f58c21377362-trusted-ca\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.300976 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzdnd\" (UniqueName: \"kubernetes.io/projected/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-api-access-kzdnd\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:58.301004 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301006 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301030 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/433c5b6b-176b-44d1-afc0-6884045e468c-trusted-ca\") pod \"console-operator-7cd56b8dbc-jk8h6\" (UID: \"433c5b6b-176b-44d1-afc0-6884045e468c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301059 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-tmp-dir\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301104 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-sysconfig\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301132 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-device-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301162 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-cni-netd\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301188 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-tmp-dir\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301224 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/449a05ee-ead2-4507-8d65-faaeb3ab3281-service-ca-bundle\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301241 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-run\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301263 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/99713fae-ae74-4da5-b435-c73edc1c4843-ovnkube-config\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301298 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-system-cni-dir\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301317 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-socket-dir-parent\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301331 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/433c5b6b-176b-44d1-afc0-6884045e468c-trusted-ca\") pod \"console-operator-7cd56b8dbc-jk8h6\" (UID: \"433c5b6b-176b-44d1-afc0-6884045e468c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301357 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/514e4bfb-77ef-4b1f-977a-3f9308cf886c-trusted-ca\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301387 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/433c5b6b-176b-44d1-afc0-6884045e468c-serving-cert\") pod \"console-operator-7cd56b8dbc-jk8h6\" (UID: \"433c5b6b-176b-44d1-afc0-6884045e468c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301428 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f594\" (UniqueName: \"kubernetes.io/projected/6de4fd5d-d10a-4630-80bf-3838fb02d679-kube-api-access-8f594\") pod \"network-check-source-6f78d59f99-qtbqg\" (UID: \"6de4fd5d-d10a-4630-80bf-3838fb02d679\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qtbqg" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301517 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/99713fae-ae74-4da5-b435-c73edc1c4843-ovnkube-config\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301533 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/433c5b6b-176b-44d1-afc0-6884045e468c-serving-cert\") pod \"console-operator-7cd56b8dbc-jk8h6\" (UID: \"433c5b6b-176b-44d1-afc0-6884045e468c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301536 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301694 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44-serving-cert\") pod \"service-ca-operator-544d749c8c-9ncj4\" (UID: \"23f64ae9-fdda-4f2e-b3a5-1c552cc55a44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301752 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-config-volume\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:58.301881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.301789 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7dde7dd-245b-4437-9464-0e8b739caa98-cni-binary-copy\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.302849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.302074 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-var-lib-kubelet\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.302849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.302086 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/514e4bfb-77ef-4b1f-977a-3f9308cf886c-trusted-ca\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:58.302849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.302121 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-run-multus-certs\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.302849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.302156 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-sys\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.302849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.302161 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2e4493e-d29f-49b6-851e-cbd01214bf5c-original-pull-secret\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:58.302849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.302212 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sx9r\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-kube-api-access-6sx9r\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:58.302849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.302259 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44-serving-cert\") pod \"service-ca-operator-544d749c8c-9ncj4\" (UID: \"23f64ae9-fdda-4f2e-b3a5-1c552cc55a44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" Nov 03 19:56:58.302849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.302374 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-config-volume\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:58.302849 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.302534 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7dde7dd-245b-4437-9464-0e8b739caa98-cni-binary-copy\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.305627 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.305602 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jhkj\" (UniqueName: \"kubernetes.io/projected/a24da6cd-d805-4f8a-ac99-89dbd45c86c3-kube-api-access-5jhkj\") pod \"iptables-alerter-fjr5q\" (UID: \"a24da6cd-d805-4f8a-ac99-89dbd45c86c3\") " pod="openshift-network-operator/iptables-alerter-fjr5q" Nov 03 19:56:58.305762 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.305741 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl89h\" (UniqueName: \"kubernetes.io/projected/158d8c01-2d31-43b4-a098-0c731c1fd163-kube-api-access-wl89h\") pod \"kas-connection-checker-685895c7c9-6r7g9\" (UID: \"158d8c01-2d31-43b4-a098-0c731c1fd163\") " pod="kube-system/kas-connection-checker-685895c7c9-6r7g9" Nov 03 19:56:58.307619 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.307599 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g8dn\" (UniqueName: \"kubernetes.io/projected/88c8909b-9acd-46d0-9f19-d74e2db51f77-kube-api-access-7g8dn\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:58.307700 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.307606 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bqcz\" (UniqueName: \"kubernetes.io/projected/26ac4e9f-3445-4994-aa1b-086b39b8c1cc-kube-api-access-7bqcz\") pod \"node-ca-jjkqt\" (UID: \"26ac4e9f-3445-4994-aa1b-086b39b8c1cc\") " pod="openshift-image-registry/node-ca-jjkqt" Nov 03 19:56:58.307983 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.307962 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6ggm\" (UniqueName: \"kubernetes.io/projected/f7dde7dd-245b-4437-9464-0e8b739caa98-kube-api-access-z6ggm\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.308015 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.307994 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh8xj\" (UniqueName: \"kubernetes.io/projected/61f1d7a9-a8b3-4db5-878d-4b8d1632fafc-kube-api-access-jh8xj\") pod \"kas-connection-checker-685895c7c9-5dgdg\" (UID: \"61f1d7a9-a8b3-4db5-878d-4b8d1632fafc\") " pod="kube-system/kas-connection-checker-685895c7c9-5dgdg" Nov 03 19:56:58.318761 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.318737 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd49x\" (UniqueName: \"kubernetes.io/projected/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-kube-api-access-dd49x\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:58.318863 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.318792 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brzfd\" (UniqueName: \"kubernetes.io/projected/ab34f1da-ace1-443d-bddd-9d9905d22e52-kube-api-access-brzfd\") pod \"network-metrics-daemon-dcqpv\" (UID: \"ab34f1da-ace1-443d-bddd-9d9905d22e52\") " pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:58.318929 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.318886 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctj28\" (UniqueName: \"kubernetes.io/projected/99713fae-ae74-4da5-b435-c73edc1c4843-kube-api-access-ctj28\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.318994 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.318974 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-bound-sa-token\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:58.319070 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.319055 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb6j8\" (UniqueName: \"kubernetes.io/projected/653cb377-919b-4793-85a4-0d60d6aeaff6-kube-api-access-vb6j8\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:58.319101 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.319066 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgswz\" (UniqueName: \"kubernetes.io/projected/0d9796ba-d26b-494e-a4b1-2ebab6496f49-kube-api-access-pgswz\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:58.319244 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.319228 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml5nd\" (UniqueName: \"kubernetes.io/projected/4726eb4c-48e8-423d-a62c-250a18bd8b83-kube-api-access-ml5nd\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:58.320309 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.320294 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hb8f\" (UniqueName: \"kubernetes.io/projected/23f64ae9-fdda-4f2e-b3a5-1c552cc55a44-kube-api-access-6hb8f\") pod \"service-ca-operator-544d749c8c-9ncj4\" (UID: \"23f64ae9-fdda-4f2e-b3a5-1c552cc55a44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" Nov 03 19:56:58.323819 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.323516 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzdnd\" (UniqueName: \"kubernetes.io/projected/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-api-access-kzdnd\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:58.323819 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.323741 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-bound-sa-token\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:58.323819 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.323779 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm2tl\" (UniqueName: \"kubernetes.io/projected/ff051e4a-95a6-4c0d-9f69-773974d6eb7c-kube-api-access-gm2tl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq\" (UID: \"ff051e4a-95a6-4c0d-9f69-773974d6eb7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" Nov 03 19:56:58.324023 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.323864 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vgq9\" (UniqueName: \"kubernetes.io/projected/c60cb950-5dd9-4a2b-bba3-b3c38d3591e3-kube-api-access-6vgq9\") pod \"volume-data-source-validator-58c9f5f46d-mvxwm\" (UID: \"c60cb950-5dd9-4a2b-bba3-b3c38d3591e3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mvxwm" Nov 03 19:56:58.324023 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.323914 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxblp\" (UniqueName: \"kubernetes.io/projected/33b6188c-0202-4bc8-813f-2d189587f3aa-kube-api-access-gxblp\") pod \"periodic-gathering-l984r-mntts\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:56:58.324023 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.323994 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk6q8\" (UniqueName: \"kubernetes.io/projected/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-kube-api-access-sk6q8\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.324164 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.324072 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkf49\" (UniqueName: \"kubernetes.io/projected/5a7d187d-9a17-475b-8ee4-ae5836d9a73a-kube-api-access-gkf49\") pod \"kas-connection-checker-685895c7c9-qkzrh\" (UID: \"5a7d187d-9a17-475b-8ee4-ae5836d9a73a\") " pod="kube-system/kas-connection-checker-685895c7c9-qkzrh" Nov 03 19:56:58.324164 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.324119 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sx9r\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-kube-api-access-6sx9r\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:58.324270 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.324227 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djg6c\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-kube-api-access-djg6c\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:58.324330 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.324311 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7vhv\" (UniqueName: \"kubernetes.io/projected/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40-kube-api-access-m7vhv\") pod \"node-resolver-ksm7q\" (UID: \"81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40\") " pod="openshift-dns/node-resolver-ksm7q" Nov 03 19:56:58.324649 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.324634 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5sh5\" (UniqueName: \"kubernetes.io/projected/aed0ee7d-d648-4138-b5ef-2ae6795321d2-kube-api-access-w5sh5\") pod \"ingress-canary-95k62\" (UID: \"aed0ee7d-d648-4138-b5ef-2ae6795321d2\") " pod="openshift-ingress-canary/ingress-canary-95k62" Nov 03 19:56:58.324680 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.324636 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtdbf\" (UniqueName: \"kubernetes.io/projected/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-kube-api-access-gtdbf\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:58.325536 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.325520 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x4lc\" (UniqueName: \"kubernetes.io/projected/433c5b6b-176b-44d1-afc0-6884045e468c-kube-api-access-8x4lc\") pod \"console-operator-7cd56b8dbc-jk8h6\" (UID: \"433c5b6b-176b-44d1-afc0-6884045e468c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:56:58.325641 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.325626 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f594\" (UniqueName: \"kubernetes.io/projected/6de4fd5d-d10a-4630-80bf-3838fb02d679-kube-api-access-8f594\") pod \"network-check-source-6f78d59f99-qtbqg\" (UID: \"6de4fd5d-d10a-4630-80bf-3838fb02d679\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qtbqg" Nov 03 19:56:58.328449 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.328430 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxkg4\" (UniqueName: \"kubernetes.io/projected/449a05ee-ead2-4507-8d65-faaeb3ab3281-kube-api-access-zxkg4\") pod \"insights-operator-7c4986d449-zl889\" (UID: \"449a05ee-ead2-4507-8d65-faaeb3ab3281\") " pod="openshift-insights/insights-operator-7c4986d449-zl889" Nov 03 19:56:58.328738 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.328723 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsb75\" (UniqueName: \"kubernetes.io/projected/f90a0a4e-2794-46fc-a799-9c1593d859ab-kube-api-access-tsb75\") pod \"network-check-target-zn2ms\" (UID: \"f90a0a4e-2794-46fc-a799-9c1593d859ab\") " pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:56:58.403478 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403453 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-run-ovn-kubernetes\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.403478 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403482 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:58.403688 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403496 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-systemd-units\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.403688 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403511 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-registration-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:58.403688 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403533 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40-hosts-file\") pod \"node-resolver-ksm7q\" (UID: \"81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40\") " pod="openshift-dns/node-resolver-ksm7q" Nov 03 19:56:58.403688 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403548 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-socket-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:58.403688 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403568 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-kubernetes\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.403688 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403571 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-run-ovn-kubernetes\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.403688 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403584 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-systemd-units\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.403688 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403568 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:58.403688 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403611 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-socket-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:58.403688 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403616 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:58.403688 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403644 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40-hosts-file\") pod \"node-resolver-ksm7q\" (UID: \"81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40\") " pod="openshift-dns/node-resolver-ksm7q" Nov 03 19:56:58.403688 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403621 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-kubernetes\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.403688 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403619 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-registration-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:58.403688 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403687 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-run-ovn\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.403688 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403660 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-run-ovn\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.403696 4918 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403717 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403735 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2zqfd\" (UID: \"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403751 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-run-openvswitch\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.403776 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls podName:4726eb4c-48e8-423d-a62c-250a18bd8b83 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:58.903762549 +0000 UTC m=+1.835630041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nb7st" (UID: "4726eb4c-48e8-423d-a62c-250a18bd8b83") : secret "cluster-monitoring-operator-tls" not found Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403778 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-run-openvswitch\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403791 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-etc-kubernetes\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.403795 4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.403807 4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d95b6bf4b-dd7kz: secret "image-registry-tls" not found Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403817 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d2e4493e-d29f-49b6-851e-cbd01214bf5c-kubelet-config\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.403827 4918 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403833 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-cni-bin\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.403852 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls podName:514e4bfb-77ef-4b1f-977a-3f9308cf886c nodeName:}" failed. No retries permitted until 2026-11-03 19:56:58.903841744 +0000 UTC m=+1.835709262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls") pod "image-registry-5d95b6bf4b-dd7kz" (UID: "514e4bfb-77ef-4b1f-977a-3f9308cf886c") : secret "image-registry-tls" not found Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.403867 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert podName:ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:58.903857799 +0000 UTC m=+1.835725291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2zqfd" (UID: "ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22") : secret "networking-console-plugin-cert" not found Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403864 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-cni-bin\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403878 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-conf-dir\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403889 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-etc-kubernetes\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403906 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-host\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403915 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d2e4493e-d29f-49b6-851e-cbd01214bf5c-kubelet-config\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403906 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-conf-dir\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403935 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-host\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403933 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-node-log\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403958 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-node-log\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403961 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-log-socket\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403984 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404001 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-etc-openvswitch\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.403986 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-log-socket\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404025 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-etc-openvswitch\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404043 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-var-lib-cni-multus\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.404052 4918 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404060 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-kubelet\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404079 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404093 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26ac4e9f-3445-4994-aa1b-086b39b8c1cc-host\") pod \"node-ca-jjkqt\" (UID: \"26ac4e9f-3445-4994-aa1b-086b39b8c1cc\") " pod="openshift-image-registry/node-ca-jjkqt" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404097 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-var-lib-cni-multus\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.404110 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls podName:93a3cf28-8c0c-4735-b177-d28bc7c8ea3f nodeName:}" failed. No retries permitted until 2026-11-03 19:56:58.904094803 +0000 UTC m=+1.835962309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls") pod "dns-default-xsh7l" (UID: "93a3cf28-8c0c-4735-b177-d28bc7c8ea3f") : secret "dns-default-metrics-tls" not found Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404111 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26ac4e9f-3445-4994-aa1b-086b39b8c1cc-host\") pod \"node-ca-jjkqt\" (UID: \"26ac4e9f-3445-4994-aa1b-086b39b8c1cc\") " pod="openshift-image-registry/node-ca-jjkqt" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404124 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-kubelet\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404141 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a24da6cd-d805-4f8a-ac99-89dbd45c86c3-host-slash\") pod \"iptables-alerter-fjr5q\" (UID: \"a24da6cd-d805-4f8a-ac99-89dbd45c86c3\") " pod="openshift-network-operator/iptables-alerter-fjr5q" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404171 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404221 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a24da6cd-d805-4f8a-ac99-89dbd45c86c3-host-slash\") pod \"iptables-alerter-fjr5q\" (UID: \"a24da6cd-d805-4f8a-ac99-89dbd45c86c3\") " pod="openshift-network-operator/iptables-alerter-fjr5q" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404231 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-modprobe-d\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.404253 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle podName:653cb377-919b-4793-85a4-0d60d6aeaff6 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:58.904239009 +0000 UTC m=+1.836106507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle") pod "router-default-6bfcd89b85-9qb4s" (UID: "653cb377-919b-4793-85a4-0d60d6aeaff6") : configmap references non-existent config key: service-ca.crt Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.404252 4918 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404276 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs\") pod \"network-metrics-daemon-dcqpv\" (UID: \"ab34f1da-ace1-443d-bddd-9d9905d22e52\") " pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.404297 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm podName:1fc8cec5-9248-4a47-822e-d81b37a380ef nodeName:}" failed. No retries permitted until 2026-11-03 19:56:58.904284393 +0000 UTC m=+1.836151923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k9x2f" (UID: "1fc8cec5-9248-4a47-822e-d81b37a380ef") : configmap "kube-rbac-proxy" not found Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404332 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404338 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-modprobe-d\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.404338 4918 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.404380 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs podName:ab34f1da-ace1-443d-bddd-9d9905d22e52 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:58.904371942 +0000 UTC m=+1.836239435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs") pod "network-metrics-daemon-dcqpv" (UID: "ab34f1da-ace1-443d-bddd-9d9905d22e52") : secret "metrics-daemon-secret" not found Nov 03 19:56:58.404353 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404358 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-var-lib-cni-bin\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.404390 4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404415 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-systemd\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.404427 4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9d99c5c47-ttssq: secret "image-registry-tls" not found Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404441 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-var-lib-openvswitch\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404454 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-systemd\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404466 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-var-lib-kubelet\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404415 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-var-lib-cni-bin\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.404472 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls podName:adc4cee9-00f0-4fac-b655-f58c21377362 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:58.904461413 +0000 UTC m=+1.836328931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls") pod "image-registry-9d99c5c47-ttssq" (UID: "adc4cee9-00f0-4fac-b655-f58c21377362") : secret "image-registry-tls" not found Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404493 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-var-lib-openvswitch\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404518 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-run-systemd\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404525 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-var-lib-kubelet\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404535 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-etc-selinux\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404553 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-run-netns\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404561 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-run-systemd\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404566 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-lib-modules\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404598 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-cni-dir\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404602 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-run-netns\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404630 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1fc8cec5-9248-4a47-822e-d81b37a380ef-crio-socket\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404644 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-cni-dir\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404647 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404649 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-lib-modules\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404654 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-etc-selinux\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404672 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-sysconfig\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404687 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-device-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404692 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1fc8cec5-9248-4a47-822e-d81b37a380ef-crio-socket\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.404697 4918 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404701 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-cni-netd\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404715 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-run\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404724 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-device-dir\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404730 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-sysconfig\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.404737 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls podName:fcfef4da-a4d7-4ef0-a383-ff2acb8cac79 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:58.9047251 +0000 UTC m=+1.836592608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fsrm8" (UID: "fcfef4da-a4d7-4ef0-a383-ff2acb8cac79") : secret "samples-operator-tls" not found Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404756 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-cni-netd\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404764 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-system-cni-dir\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404783 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-socket-dir-parent\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404807 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-var-lib-kubelet\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404747 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-run\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404841 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-system-cni-dir\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404851 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-multus-socket-dir-parent\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404835 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-run-multus-certs\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404855 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-var-lib-kubelet\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404886 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-sys\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404905 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-slash\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404920 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-hostroot\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404922 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-run-multus-certs\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404939 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-sys\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404952 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-hostroot\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404974 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-slash\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.404990 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405019 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-sysctl-d\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.405036 4918 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405039 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88c8909b-9acd-46d0-9f19-d74e2db51f77-system-cni-dir\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405055 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-cnibin\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.405062 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs podName:653cb377-919b-4793-85a4-0d60d6aeaff6 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:58.905054701 +0000 UTC m=+1.836922208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs") pod "router-default-6bfcd89b85-9qb4s" (UID: "653cb377-919b-4793-85a4-0d60d6aeaff6") : secret "router-metrics-certs-default" not found Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405081 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88c8909b-9acd-46d0-9f19-d74e2db51f77-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405089 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-cnibin\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405100 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405114 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88c8909b-9acd-46d0-9f19-d74e2db51f77-system-cni-dir\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405121 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405133 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88c8909b-9acd-46d0-9f19-d74e2db51f77-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405138 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-run-netns\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405143 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405163 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99713fae-ae74-4da5-b435-c73edc1c4843-host-run-netns\") pod \"ovnkube-node-bvlzl\" (UID: \"99713fae-ae74-4da5-b435-c73edc1c4843\") " pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405189 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88c8909b-9acd-46d0-9f19-d74e2db51f77-cnibin\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405191 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-sysctl-d\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405205 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d2e4493e-d29f-49b6-851e-cbd01214bf5c-dbus\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405219 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88c8909b-9acd-46d0-9f19-d74e2db51f77-cnibin\") pod \"multus-additional-cni-plugins-wjgg6\" (UID: \"88c8909b-9acd-46d0-9f19-d74e2db51f77\") " pod="openshift-multus/multus-additional-cni-plugins-wjgg6" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405224 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-sys-fs\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405245 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-run-k8s-cni-cncf-io\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405254 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d2e4493e-d29f-49b6-851e-cbd01214bf5c-dbus\") pod \"global-pull-secret-syncer-mvsdz\" (UID: \"d2e4493e-d29f-49b6-851e-cbd01214bf5c\") " pod="kube-system/global-pull-secret-syncer-mvsdz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405261 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-sysctl-conf\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405275 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0d9796ba-d26b-494e-a4b1-2ebab6496f49-sys-fs\") pod \"aws-ebs-csi-driver-node-dgmfk\" (UID: \"0d9796ba-d26b-494e-a4b1-2ebab6496f49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgmfk" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.405228 4918 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405347 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f838cffc-2fc7-4a3f-880b-8a80298cb1b4-etc-sysctl-conf\") pod \"tuned-5vpvg\" (UID: \"f838cffc-2fc7-4a3f-880b-8a80298cb1b4\") " pod="openshift-cluster-node-tuning-operator/tuned-5vpvg" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.405375 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7dde7dd-245b-4437-9464-0e8b739caa98-host-run-k8s-cni-cncf-io\") pod \"multus-9gbmz\" (UID: \"f7dde7dd-245b-4437-9464-0e8b739caa98\") " pod="openshift-multus/multus-9gbmz" Nov 03 19:56:58.405961 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.405380 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls podName:1fc8cec5-9248-4a47-822e-d81b37a380ef nodeName:}" failed. No retries permitted until 2026-11-03 19:56:58.905371295 +0000 UTC m=+1.837238800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k9x2f" (UID: "1fc8cec5-9248-4a47-822e-d81b37a380ef") : secret "insights-runtime-extractor-tls" not found Nov 03 19:56:58.479686 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.479623 4918 scope.go:122] "RemoveContainer" containerID="33c82892c5f4b399f0d1f3ee43a5aee3b52b13b00868cd4bbdb8ac251bc91a2f" Nov 03 19:56:58.908924 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.908891 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:58.908924 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.908930 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:58.909160 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.908949 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2zqfd\" (UID: \"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:56:58.909160 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.908975 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:58.909160 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.908995 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:58.909160 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.909012 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:58.909160 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.909028 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs\") pod \"network-metrics-daemon-dcqpv\" (UID: \"ab34f1da-ace1-443d-bddd-9d9905d22e52\") " pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:58.909160 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.909047 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:58.909160 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909048 4918 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:56:58.909160 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.909088 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:58.909160 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909098 4918 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 19:56:58.909160 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909121 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert podName:ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:59.90910261 +0000 UTC m=+2.840970116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2zqfd" (UID: "ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22") : secret "networking-console-plugin-cert" not found Nov 03 19:56:58.909160 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909146 4918 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:56:58.909160 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909153 4918 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:56:58.909619 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909098 4918 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 19:56:58.909619 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909216 4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:56:58.909619 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909155 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls podName:4726eb4c-48e8-423d-a62c-250a18bd8b83 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:59.909139459 +0000 UTC m=+2.841006955 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nb7st" (UID: "4726eb4c-48e8-423d-a62c-250a18bd8b83") : secret "cluster-monitoring-operator-tls" not found Nov 03 19:56:58.909619 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909229 4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9d99c5c47-ttssq: secret "image-registry-tls" not found Nov 03 19:56:58.909619 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909054 4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:56:58.909619 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909250 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle podName:653cb377-919b-4793-85a4-0d60d6aeaff6 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:59.909237232 +0000 UTC m=+2.841104725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle") pod "router-default-6bfcd89b85-9qb4s" (UID: "653cb377-919b-4793-85a4-0d60d6aeaff6") : configmap references non-existent config key: service-ca.crt Nov 03 19:56:58.909619 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909252 4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d95b6bf4b-dd7kz: secret "image-registry-tls" not found Nov 03 19:56:58.909619 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909275 4918 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 19:56:58.909619 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909295 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls podName:93a3cf28-8c0c-4735-b177-d28bc7c8ea3f nodeName:}" failed. No retries permitted until 2026-11-03 19:56:59.909281663 +0000 UTC m=+2.841149175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls") pod "dns-default-xsh7l" (UID: "93a3cf28-8c0c-4735-b177-d28bc7c8ea3f") : secret "dns-default-metrics-tls" not found Nov 03 19:56:58.909619 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.909313 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:58.909619 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909322 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs podName:ab34f1da-ace1-443d-bddd-9d9905d22e52 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:59.90930763 +0000 UTC m=+2.841175133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs") pod "network-metrics-daemon-dcqpv" (UID: "ab34f1da-ace1-443d-bddd-9d9905d22e52") : secret "metrics-daemon-secret" not found Nov 03 19:56:58.909619 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909351 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls podName:fcfef4da-a4d7-4ef0-a383-ff2acb8cac79 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:59.909340067 +0000 UTC m=+2.841207569 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fsrm8" (UID: "fcfef4da-a4d7-4ef0-a383-ff2acb8cac79") : secret "samples-operator-tls" not found Nov 03 19:56:58.909619 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909367 4918 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 19:56:58.909619 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909378 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm podName:1fc8cec5-9248-4a47-822e-d81b37a380ef nodeName:}" failed. No retries permitted until 2026-11-03 19:56:59.90936167 +0000 UTC m=+2.841229174 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k9x2f" (UID: "1fc8cec5-9248-4a47-822e-d81b37a380ef") : configmap "kube-rbac-proxy" not found Nov 03 19:56:58.909619 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909412 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs podName:653cb377-919b-4793-85a4-0d60d6aeaff6 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:59.909402018 +0000 UTC m=+2.841269513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs") pod "router-default-6bfcd89b85-9qb4s" (UID: "653cb377-919b-4793-85a4-0d60d6aeaff6") : secret "router-metrics-certs-default" not found Nov 03 19:56:58.909619 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909432 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls podName:adc4cee9-00f0-4fac-b655-f58c21377362 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:59.909422776 +0000 UTC m=+2.841290265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls") pod "image-registry-9d99c5c47-ttssq" (UID: "adc4cee9-00f0-4fac-b655-f58c21377362") : secret "image-registry-tls" not found Nov 03 19:56:58.909619 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909457 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls podName:514e4bfb-77ef-4b1f-977a-3f9308cf886c nodeName:}" failed. No retries permitted until 2026-11-03 19:56:59.909452042 +0000 UTC m=+2.841319530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls") pod "image-registry-5d95b6bf4b-dd7kz" (UID: "514e4bfb-77ef-4b1f-977a-3f9308cf886c") : secret "image-registry-tls" not found Nov 03 19:56:58.909619 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:58.909471 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:58.909619 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909570 4918 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:56:58.909619 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:58.909599 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls podName:1fc8cec5-9248-4a47-822e-d81b37a380ef nodeName:}" failed. No retries permitted until 2026-11-03 19:56:59.909592395 +0000 UTC m=+2.841459884 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k9x2f" (UID: "1fc8cec5-9248-4a47-822e-d81b37a380ef") : secret "insights-runtime-extractor-tls" not found Nov 03 19:56:59.264492 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.264422 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 19:56:59.264881 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.264865 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/1.log" Nov 03 19:56:59.264919 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.264898 4918 generic.go:356] "Generic (PLEG): container finished" podID="433c5b6b-176b-44d1-afc0-6884045e468c" containerID="c392ec8f13457c33d7f1530212f221d2ce0247ab66a61310e9747d1f74526da9" exitCode=255 Nov 03 19:56:59.264995 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.264975 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" event={"ID":"433c5b6b-176b-44d1-afc0-6884045e468c","Type":"ContainerDied","Data":"c392ec8f13457c33d7f1530212f221d2ce0247ab66a61310e9747d1f74526da9"} Nov 03 19:56:59.265043 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.265031 4918 scope.go:122] "RemoveContainer" containerID="33c82892c5f4b399f0d1f3ee43a5aee3b52b13b00868cd4bbdb8ac251bc91a2f" Nov 03 19:56:59.265422 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.265398 4918 scope.go:122] "RemoveContainer" containerID="c392ec8f13457c33d7f1530212f221d2ce0247ab66a61310e9747d1f74526da9" Nov 03 19:56:59.265599 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:59.265581 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jk8h6_openshift-console-operator(433c5b6b-176b-44d1-afc0-6884045e468c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" podUID="433c5b6b-176b-44d1-afc0-6884045e468c" Nov 03 19:56:59.583900 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.583863 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ntz8w" Nov 03 19:56:59.920014 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.919934 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:59.920014 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.919973 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:59.920014 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.919998 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs\") pod \"network-metrics-daemon-dcqpv\" (UID: \"ab34f1da-ace1-443d-bddd-9d9905d22e52\") " pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:56:59.920288 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.920028 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:59.920288 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.920069 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:56:59.920288 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:59.920095 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle podName:653cb377-919b-4793-85a4-0d60d6aeaff6 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:01.920077299 +0000 UTC m=+4.851944788 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle") pod "router-default-6bfcd89b85-9qb4s" (UID: "653cb377-919b-4793-85a4-0d60d6aeaff6") : configmap references non-existent config key: service-ca.crt Nov 03 19:56:59.920288 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:59.920143 4918 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:56:59.920288 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.920158 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:56:59.920288 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.920212 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:59.920288 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:59.920228 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls podName:fcfef4da-a4d7-4ef0-a383-ff2acb8cac79 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:01.920210237 +0000 UTC m=+4.852077743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fsrm8" (UID: "fcfef4da-a4d7-4ef0-a383-ff2acb8cac79") : secret "samples-operator-tls" not found Nov 03 19:56:59.920288 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:59.920143 4918 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 19:56:59.920288 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:59.920265 4918 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 19:56:59.920288 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:59.920277 4918 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:56:59.920288 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.920291 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:59.920784 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:59.920310 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs podName:ab34f1da-ace1-443d-bddd-9d9905d22e52 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:01.920290126 +0000 UTC m=+4.852157641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs") pod "network-metrics-daemon-dcqpv" (UID: "ab34f1da-ace1-443d-bddd-9d9905d22e52") : secret "metrics-daemon-secret" not found Nov 03 19:56:59.920784 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:59.920342 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls podName:1fc8cec5-9248-4a47-822e-d81b37a380ef nodeName:}" failed. No retries permitted until 2026-11-03 19:57:01.920326819 +0000 UTC m=+4.852194308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k9x2f" (UID: "1fc8cec5-9248-4a47-822e-d81b37a380ef") : secret "insights-runtime-extractor-tls" not found Nov 03 19:56:59.920784 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:59.920366 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs podName:653cb377-919b-4793-85a4-0d60d6aeaff6 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:01.920354547 +0000 UTC m=+4.852222036 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs") pod "router-default-6bfcd89b85-9qb4s" (UID: "653cb377-919b-4793-85a4-0d60d6aeaff6") : secret "router-metrics-certs-default" not found Nov 03 19:56:59.920784 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.920396 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:59.920784 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.920431 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2zqfd\" (UID: \"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:56:59.920784 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.920464 4918 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 19:56:59.920784 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.920482 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:56:59.920784 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:59.920576 4918 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:56:59.920784 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:59.920582 4918 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:56:59.920784 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.920615 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fc8cec5-9248-4a47-822e-d81b37a380ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:56:59.920784 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:59.920641 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert podName:ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:01.920626062 +0000 UTC m=+4.852493565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2zqfd" (UID: "ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22") : secret "networking-console-plugin-cert" not found Nov 03 19:56:59.920784 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:56:59.920661 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls podName:93a3cf28-8c0c-4735-b177-d28bc7c8ea3f nodeName:}" failed. No retries permitted until 2026-11-03 19:57:01.920651282 +0000 UTC m=+4.852518784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls") pod "dns-default-xsh7l" (UID: "93a3cf28-8c0c-4735-b177-d28bc7c8ea3f") : secret "dns-default-metrics-tls" not found Nov 03 19:56:59.922685 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.922664 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4726eb4c-48e8-423d-a62c-250a18bd8b83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nb7st\" (UID: \"4726eb4c-48e8-423d-a62c-250a18bd8b83\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:59.922789 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.922684 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls\") pod \"image-registry-9d99c5c47-ttssq\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:59.922789 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.922666 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls\") pod \"image-registry-5d95b6bf4b-dd7kz\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:56:59.980855 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.980820 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" Nov 03 19:56:59.983007 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.982987 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:56:59.990400 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:56:59.990380 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:57:00.265059 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:00.264984 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st"] Nov 03 19:57:00.268276 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:57:00.268245 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4726eb4c_48e8_423d_a62c_250a18bd8b83.slice/crio-2801732f1e6907ea724508da2b76c0a30fb86cfe7c8c85c86b1e76f2bfa84c15 WatchSource:0}: Error finding container 2801732f1e6907ea724508da2b76c0a30fb86cfe7c8c85c86b1e76f2bfa84c15: Status 404 returned error can't find the container with id 2801732f1e6907ea724508da2b76c0a30fb86cfe7c8c85c86b1e76f2bfa84c15 Nov 03 19:57:00.268484 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:00.268466 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9d99c5c47-ttssq"] Nov 03 19:57:00.271231 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:00.271205 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d95b6bf4b-dd7kz"] Nov 03 19:57:00.271467 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:00.271450 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 19:57:00.271591 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:00.271570 4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:57:00.272625 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:57:00.272603 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc4cee9_00f0_4fac_b655_f58c21377362.slice/crio-8327382533dc4728e498c02f706cd469101a79ec1aaa2579372bbb24e180b893 WatchSource:0}: Error finding container 8327382533dc4728e498c02f706cd469101a79ec1aaa2579372bbb24e180b893: Status 404 returned error can't find the container with id 8327382533dc4728e498c02f706cd469101a79ec1aaa2579372bbb24e180b893 Nov 03 19:57:00.274189 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:57:00.274162 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514e4bfb_77ef_4b1f_977a_3f9308cf886c.slice/crio-d9837cebc843f195a9d4d3c8f65191fef8d20657f1a8ec367dcb5f4783de7716 WatchSource:0}: Error finding container d9837cebc843f195a9d4d3c8f65191fef8d20657f1a8ec367dcb5f4783de7716: Status 404 returned error can't find the container with id d9837cebc843f195a9d4d3c8f65191fef8d20657f1a8ec367dcb5f4783de7716 Nov 03 19:57:01.277479 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:01.277432 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" event={"ID":"514e4bfb-77ef-4b1f-977a-3f9308cf886c","Type":"ContainerStarted","Data":"bcd99734f750dfcfafa967e0a20102f5771824b4430adbbbc5559882a43b7ab3"} Nov 03 19:57:01.277946 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:01.277507 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" event={"ID":"514e4bfb-77ef-4b1f-977a-3f9308cf886c","Type":"ContainerStarted","Data":"d9837cebc843f195a9d4d3c8f65191fef8d20657f1a8ec367dcb5f4783de7716"} Nov 03 19:57:01.279621 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:01.279591 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" event={"ID":"adc4cee9-00f0-4fac-b655-f58c21377362","Type":"ContainerStarted","Data":"b4a6efe2c227375e934add2ede1e00719ad2d2752368f2ae340ac5d897c4259b"} Nov 03 19:57:01.279756 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:01.279650 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" event={"ID":"adc4cee9-00f0-4fac-b655-f58c21377362","Type":"ContainerStarted","Data":"8327382533dc4728e498c02f706cd469101a79ec1aaa2579372bbb24e180b893"} Nov 03 19:57:01.280883 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:01.280859 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" event={"ID":"4726eb4c-48e8-423d-a62c-250a18bd8b83","Type":"ContainerStarted","Data":"2801732f1e6907ea724508da2b76c0a30fb86cfe7c8c85c86b1e76f2bfa84c15"} Nov 03 19:57:01.432102 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:01.432070 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:57:01.940852 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:01.940811 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2zqfd\" (UID: \"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:57:01.941026 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:01.940867 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:57:01.941026 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:01.940904 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:57:01.941026 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:01.940939 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs\") pod \"network-metrics-daemon-dcqpv\" (UID: \"ab34f1da-ace1-443d-bddd-9d9905d22e52\") " pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:57:01.941026 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:01.940975 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:57:01.941026 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:01.941004 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:57:01.941365 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:01.940937 4918 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:57:01.941365 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:01.941055 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle podName:653cb377-919b-4793-85a4-0d60d6aeaff6 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:05.941032294 +0000 UTC m=+8.872899799 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle") pod "router-default-6bfcd89b85-9qb4s" (UID: "653cb377-919b-4793-85a4-0d60d6aeaff6") : configmap references non-existent config key: service-ca.crt Nov 03 19:57:01.941365 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:01.941089 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert podName:ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:05.941077109 +0000 UTC m=+8.872944613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2zqfd" (UID: "ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22") : secret "networking-console-plugin-cert" not found Nov 03 19:57:01.941365 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:01.941089 4918 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:57:01.941365 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:01.941029 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:57:01.941365 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:01.941104 4918 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:57:01.941365 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:01.941160 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls podName:fcfef4da-a4d7-4ef0-a383-ff2acb8cac79 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:05.941148512 +0000 UTC m=+8.873016001 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fsrm8" (UID: "fcfef4da-a4d7-4ef0-a383-ff2acb8cac79") : secret "samples-operator-tls" not found Nov 03 19:57:01.941365 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:01.941027 4918 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:57:01.941365 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:01.941199 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls podName:1fc8cec5-9248-4a47-822e-d81b37a380ef nodeName:}" failed. No retries permitted until 2026-11-03 19:57:05.941188916 +0000 UTC m=+8.873056419 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k9x2f" (UID: "1fc8cec5-9248-4a47-822e-d81b37a380ef") : secret "insights-runtime-extractor-tls" not found Nov 03 19:57:01.941365 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:01.941232 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls podName:93a3cf28-8c0c-4735-b177-d28bc7c8ea3f nodeName:}" failed. No retries permitted until 2026-11-03 19:57:05.941222736 +0000 UTC m=+8.873090242 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls") pod "dns-default-xsh7l" (UID: "93a3cf28-8c0c-4735-b177-d28bc7c8ea3f") : secret "dns-default-metrics-tls" not found Nov 03 19:57:01.941365 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:01.941105 4918 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 19:57:01.941365 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:01.941265 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs podName:653cb377-919b-4793-85a4-0d60d6aeaff6 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:05.941258782 +0000 UTC m=+8.873126284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs") pod "router-default-6bfcd89b85-9qb4s" (UID: "653cb377-919b-4793-85a4-0d60d6aeaff6") : secret "router-metrics-certs-default" not found Nov 03 19:57:01.943936 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:01.943910 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab34f1da-ace1-443d-bddd-9d9905d22e52-metrics-certs\") pod \"network-metrics-daemon-dcqpv\" (UID: \"ab34f1da-ace1-443d-bddd-9d9905d22e52\") " pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:57:02.086428 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:02.086391 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dcqpv" Nov 03 19:57:02.285589 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:02.285566 4918 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:57:02.286017 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:02.285559 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nb7st" event={"ID":"4726eb4c-48e8-423d-a62c-250a18bd8b83","Type":"ContainerStarted","Data":"af7585787b929805e14f08931551b10920e1f35ea5b16df84f37cad0e28fc2b2"} Nov 03 19:57:02.286017 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:02.285780 4918 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:57:02.365045 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:02.365015 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dcqpv"] Nov 03 19:57:02.368448 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:57:02.368423 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab34f1da_ace1_443d_bddd_9d9905d22e52.slice/crio-2d09a6ac03796268917fe78439c40f0fdcfc284cfca8ab9a4f0135ed92a426cc WatchSource:0}: Error finding container 2d09a6ac03796268917fe78439c40f0fdcfc284cfca8ab9a4f0135ed92a426cc: Status 404 returned error can't find the container with id 2d09a6ac03796268917fe78439c40f0fdcfc284cfca8ab9a4f0135ed92a426cc Nov 03 19:57:02.753663 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:02.753628 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:57:02.754026 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:02.754007 4918 scope.go:122] "RemoveContainer" containerID="c392ec8f13457c33d7f1530212f221d2ce0247ab66a61310e9747d1f74526da9" Nov 03 19:57:02.754240 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:02.754219 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jk8h6_openshift-console-operator(433c5b6b-176b-44d1-afc0-6884045e468c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" podUID="433c5b6b-176b-44d1-afc0-6884045e468c" Nov 03 19:57:03.292083 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:03.292044 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dcqpv" event={"ID":"ab34f1da-ace1-443d-bddd-9d9905d22e52","Type":"ContainerStarted","Data":"2d09a6ac03796268917fe78439c40f0fdcfc284cfca8ab9a4f0135ed92a426cc"} Nov 03 19:57:03.385893 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:03.385860 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:57:03.386069 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:03.386056 4918 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:57:04.141464 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:04.141427 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:57:04.240131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:04.240093 4918 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:57:04.240528 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:04.240512 4918 scope.go:122] "RemoveContainer" containerID="c392ec8f13457c33d7f1530212f221d2ce0247ab66a61310e9747d1f74526da9" Nov 03 19:57:04.240709 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:04.240690 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jk8h6_openshift-console-operator(433c5b6b-176b-44d1-afc0-6884045e468c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" podUID="433c5b6b-176b-44d1-afc0-6884045e468c" Nov 03 19:57:04.297250 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:04.297209 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dcqpv" event={"ID":"ab34f1da-ace1-443d-bddd-9d9905d22e52","Type":"ContainerStarted","Data":"1fcadd1106e1f8637c5ddb8f958a4b08936522f73173ea03a5692bf6280dd758"} Nov 03 19:57:04.297601 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:04.297277 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dcqpv" event={"ID":"ab34f1da-ace1-443d-bddd-9d9905d22e52","Type":"ContainerStarted","Data":"d8473b6121d685ca5e4b185ea116737870e696b35a573d6ba71f70a8faa59217"} Nov 03 19:57:04.810349 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:04.810311 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:57:05.976756 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:05.976715 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:57:05.982667 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:05.982640 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2zqfd\" (UID: \"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:57:05.982791 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:05.982696 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:57:05.982791 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:05.982727 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:57:05.982791 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:05.982777 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:57:05.982791 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:05.982780 4918 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:57:05.982997 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:05.982817 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:57:05.982997 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:05.982838 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert podName:ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:13.982824499 +0000 UTC m=+16.914692001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2zqfd" (UID: "ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22") : secret "networking-console-plugin-cert" not found Nov 03 19:57:05.982997 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:05.982869 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:57:05.982997 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:05.982888 4918 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 19:57:05.982997 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:05.982890 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle podName:653cb377-919b-4793-85a4-0d60d6aeaff6 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:13.98287405 +0000 UTC m=+16.914741561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle") pod "router-default-6bfcd89b85-9qb4s" (UID: "653cb377-919b-4793-85a4-0d60d6aeaff6") : configmap references non-existent config key: service-ca.crt Nov 03 19:57:05.982997 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:05.982939 4918 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:57:05.982997 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:05.982948 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs podName:653cb377-919b-4793-85a4-0d60d6aeaff6 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:13.982934117 +0000 UTC m=+16.914801613 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs") pod "router-default-6bfcd89b85-9qb4s" (UID: "653cb377-919b-4793-85a4-0d60d6aeaff6") : secret "router-metrics-certs-default" not found Nov 03 19:57:05.982997 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:05.982987 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls podName:fcfef4da-a4d7-4ef0-a383-ff2acb8cac79 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:13.982975014 +0000 UTC m=+16.914842506 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fsrm8" (UID: "fcfef4da-a4d7-4ef0-a383-ff2acb8cac79") : secret "samples-operator-tls" not found Nov 03 19:57:05.983285 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:05.983018 4918 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:57:05.983285 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:05.983065 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls podName:1fc8cec5-9248-4a47-822e-d81b37a380ef nodeName:}" failed. No retries permitted until 2026-11-03 19:57:13.98305229 +0000 UTC m=+16.914919793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k9x2f" (UID: "1fc8cec5-9248-4a47-822e-d81b37a380ef") : secret "insights-runtime-extractor-tls" not found Nov 03 19:57:05.984963 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:05.984934 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a3cf28-8c0c-4735-b177-d28bc7c8ea3f-metrics-tls\") pod \"dns-default-xsh7l\" (UID: \"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f\") " pod="openshift-dns/dns-default-xsh7l" Nov 03 19:57:05.991080 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:05.991059 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xsh7l" Nov 03 19:57:06.265318 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:06.265294 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xsh7l"] Nov 03 19:57:06.267916 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:57:06.267889 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a3cf28_8c0c_4735_b177_d28bc7c8ea3f.slice/crio-34c83fc6ef18aa79b4c28340bb8e5b69986f76ee61df9180e006cb2f547ba39f WatchSource:0}: Error finding container 34c83fc6ef18aa79b4c28340bb8e5b69986f76ee61df9180e006cb2f547ba39f: Status 404 returned error can't find the container with id 34c83fc6ef18aa79b4c28340bb8e5b69986f76ee61df9180e006cb2f547ba39f Nov 03 19:57:06.307965 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:06.307931 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xsh7l" event={"ID":"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f","Type":"ContainerStarted","Data":"34c83fc6ef18aa79b4c28340bb8e5b69986f76ee61df9180e006cb2f547ba39f"} Nov 03 19:57:08.316628 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:08.316602 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xsh7l" event={"ID":"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f","Type":"ContainerStarted","Data":"6ea458d9d2933505e1481c7ec42eae8826a9ade41637f4ecf41605ed65e27aa3"} Nov 03 19:57:09.321281 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:09.321243 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xsh7l" event={"ID":"93a3cf28-8c0c-4735-b177-d28bc7c8ea3f","Type":"ContainerStarted","Data":"36ef6e57a75fb1c2d09609fabf1c1cbead8de71183699c86dcbbef30c372b1ec"} Nov 03 19:57:09.321677 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:09.321384 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xsh7l" Nov 03 19:57:10.559442 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:10.559410 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bvlzl" Nov 03 19:57:11.207693 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:11.207657 4918 scope.go:122] "RemoveContainer" containerID="c392ec8f13457c33d7f1530212f221d2ce0247ab66a61310e9747d1f74526da9" Nov 03 19:57:11.331082 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:11.331051 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 19:57:11.331280 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:11.331161 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" event={"ID":"433c5b6b-176b-44d1-afc0-6884045e468c","Type":"ContainerStarted","Data":"f46e2be6174d61d2650d20d9284d580d4906ed9d03ce85329ebcf6646c2cea17"} Nov 03 19:57:11.331681 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:11.331505 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:57:11.528208 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:11.528121 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jk8h6" Nov 03 19:57:11.724588 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:11.724553 4918 patch_prober.go:29] interesting pod/image-registry-5d95b6bf4b-dd7kz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 19:57:11.724955 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:11.724611 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" podUID="514e4bfb-77ef-4b1f-977a-3f9308cf886c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:57:13.838318 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:13.838257 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9d99c5c47-ttssq"] Nov 03 19:57:13.839015 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:13.838986 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2xzzj"] Nov 03 19:57:13.842738 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:13.842717 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2xzzj" Nov 03 19:57:13.853995 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:13.853962 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2xzzj"] Nov 03 19:57:13.858304 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:13.858261 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 19:57:13.858521 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:13.858346 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 19:57:13.933481 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:13.933450 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6pnl5\"" Nov 03 19:57:13.953131 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:13.953102 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46ktp\" (UniqueName: \"kubernetes.io/projected/57f58b4a-baf9-4fb1-a4a4-d2383b98b9d5-kube-api-access-46ktp\") pod \"migrator-66df5d496b-2xzzj\" (UID: \"57f58b4a-baf9-4fb1-a4a4-d2383b98b9d5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2xzzj" Nov 03 19:57:14.053686 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.053651 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2zqfd\" (UID: \"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:57:14.053874 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.053702 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:57:14.053874 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.053723 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46ktp\" (UniqueName: \"kubernetes.io/projected/57f58b4a-baf9-4fb1-a4a4-d2383b98b9d5-kube-api-access-46ktp\") pod \"migrator-66df5d496b-2xzzj\" (UID: \"57f58b4a-baf9-4fb1-a4a4-d2383b98b9d5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2xzzj" Nov 03 19:57:14.053874 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.053761 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:57:14.053874 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.053790 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:57:14.053874 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.053808 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:57:14.054518 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.054486 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/653cb377-919b-4793-85a4-0d60d6aeaff6-service-ca-bundle\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:57:14.056349 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.056324 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc8cec5-9248-4a47-822e-d81b37a380ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k9x2f\" (UID: \"1fc8cec5-9248-4a47-822e-d81b37a380ef\") " pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:57:14.056467 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.056329 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2zqfd\" (UID: \"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:57:14.056728 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.056708 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653cb377-919b-4793-85a4-0d60d6aeaff6-metrics-certs\") pod \"router-default-6bfcd89b85-9qb4s\" (UID: \"653cb377-919b-4793-85a4-0d60d6aeaff6\") " pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:57:14.057014 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.056997 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcfef4da-a4d7-4ef0-a383-ff2acb8cac79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fsrm8\" (UID: \"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:57:14.065648 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.065627 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46ktp\" (UniqueName: \"kubernetes.io/projected/57f58b4a-baf9-4fb1-a4a4-d2383b98b9d5-kube-api-access-46ktp\") pod \"migrator-66df5d496b-2xzzj\" (UID: \"57f58b4a-baf9-4fb1-a4a4-d2383b98b9d5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2xzzj" Nov 03 19:57:14.076158 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.076125 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" Nov 03 19:57:14.078278 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.078262 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:57:14.083417 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.083400 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" Nov 03 19:57:14.095340 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.095316 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k9x2f" Nov 03 19:57:14.160886 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.160638 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2xzzj" Nov 03 19:57:14.366735 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.366664 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6bfcd89b85-9qb4s"] Nov 03 19:57:14.369853 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.369827 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2zqfd"] Nov 03 19:57:14.370122 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:57:14.370090 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653cb377_919b_4793_85a4_0d60d6aeaff6.slice/crio-8ada9ec55adb31c9425d720a5ea4542c39f95b864ee0b9c8b801f5f455e5fc74 WatchSource:0}: Error finding container 8ada9ec55adb31c9425d720a5ea4542c39f95b864ee0b9c8b801f5f455e5fc74: Status 404 returned error can't find the container with id 8ada9ec55adb31c9425d720a5ea4542c39f95b864ee0b9c8b801f5f455e5fc74 Nov 03 19:57:14.370570 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.370551 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8"] Nov 03 19:57:14.372839 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:57:14.372817 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1c4fd8_fa4e_45de_8c10_4e5745b0ca22.slice/crio-0525e4d8a6091aaf81493aa240444f51585c2b1529ad4518f2e1a47fbdd4dee2 WatchSource:0}: Error finding container 0525e4d8a6091aaf81493aa240444f51585c2b1529ad4518f2e1a47fbdd4dee2: Status 404 returned error can't find the container with id 0525e4d8a6091aaf81493aa240444f51585c2b1529ad4518f2e1a47fbdd4dee2 Nov 03 19:57:14.384253 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.384230 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k9x2f"] Nov 03 19:57:14.390269 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:57:14.390242 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc8cec5_9248_4a47_822e_d81b37a380ef.slice/crio-cb31d0a0494588406aa88de61a207b0298a37e7fe86695dbef79c85382e1b401 WatchSource:0}: Error finding container cb31d0a0494588406aa88de61a207b0298a37e7fe86695dbef79c85382e1b401: Status 404 returned error can't find the container with id cb31d0a0494588406aa88de61a207b0298a37e7fe86695dbef79c85382e1b401 Nov 03 19:57:14.453662 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:14.453632 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2xzzj"] Nov 03 19:57:14.456825 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:57:14.456790 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f58b4a_baf9_4fb1_a4a4_d2383b98b9d5.slice/crio-5c4072b0c12fd79cbc04d56f5cd27eae434811e8081776a2b0fc27721cee2ccf WatchSource:0}: Error finding container 5c4072b0c12fd79cbc04d56f5cd27eae434811e8081776a2b0fc27721cee2ccf: Status 404 returned error can't find the container with id 5c4072b0c12fd79cbc04d56f5cd27eae434811e8081776a2b0fc27721cee2ccf Nov 03 19:57:15.349163 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:15.349005 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k9x2f" event={"ID":"1fc8cec5-9248-4a47-822e-d81b37a380ef","Type":"ContainerStarted","Data":"2b3b6e740d2c5f11e9d435585caae02f58975f0e200603aeee627d299aaf4522"} Nov 03 19:57:15.349163 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:15.349080 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k9x2f" event={"ID":"1fc8cec5-9248-4a47-822e-d81b37a380ef","Type":"ContainerStarted","Data":"9f26c779769b89760d643a9cabab1349400321b763b7c2015e283f33f5e37438"} Nov 03 19:57:15.349163 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:15.349101 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k9x2f" event={"ID":"1fc8cec5-9248-4a47-822e-d81b37a380ef","Type":"ContainerStarted","Data":"cb31d0a0494588406aa88de61a207b0298a37e7fe86695dbef79c85382e1b401"} Nov 03 19:57:15.353084 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:15.352164 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" event={"ID":"653cb377-919b-4793-85a4-0d60d6aeaff6","Type":"ContainerStarted","Data":"bdc79107939b73936f884aacf879c38101eaa84b00c08896c4b0023295d54d82"} Nov 03 19:57:15.353084 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:15.352235 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" event={"ID":"653cb377-919b-4793-85a4-0d60d6aeaff6","Type":"ContainerStarted","Data":"8ada9ec55adb31c9425d720a5ea4542c39f95b864ee0b9c8b801f5f455e5fc74"} Nov 03 19:57:15.356889 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:15.356852 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2xzzj" event={"ID":"57f58b4a-baf9-4fb1-a4a4-d2383b98b9d5","Type":"ContainerStarted","Data":"5c4072b0c12fd79cbc04d56f5cd27eae434811e8081776a2b0fc27721cee2ccf"} Nov 03 19:57:15.359216 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:15.359152 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" event={"ID":"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79","Type":"ContainerStarted","Data":"5738c46d87a06913b0e8672a9f22e13fea8cf4d440b3c07c5005fa6771bdbd59"} Nov 03 19:57:15.360775 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:15.360748 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" event={"ID":"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22","Type":"ContainerStarted","Data":"0525e4d8a6091aaf81493aa240444f51585c2b1529ad4518f2e1a47fbdd4dee2"} Nov 03 19:57:16.078708 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:16.078664 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:57:16.081689 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:16.081660 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:57:16.369049 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:16.368583 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2xzzj" event={"ID":"57f58b4a-baf9-4fb1-a4a4-d2383b98b9d5","Type":"ContainerStarted","Data":"bff5b49eefef04f25122a5389848d817a9943e6cfdbe1dc67a5d40ddd2220afe"} Nov 03 19:57:16.371387 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:16.371349 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" event={"ID":"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79","Type":"ContainerStarted","Data":"0cd3e851ef420fb9fffe77b2c8bbadbff9f440008a60d58871c6e6e999966966"} Nov 03 19:57:16.371526 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:16.371410 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fsrm8" event={"ID":"fcfef4da-a4d7-4ef0-a383-ff2acb8cac79","Type":"ContainerStarted","Data":"7580fc6d63918bbaaf08e073214296630942964cdf5787536b7b54c6a47372bd"} Nov 03 19:57:16.373226 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:16.373201 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-2zqfd" event={"ID":"ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22","Type":"ContainerStarted","Data":"4395f29ce44b4bdb38da3ffcc5316364c6c8c60dfdc41431cb56446a59f9718f"} Nov 03 19:57:16.374269 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:16.373526 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:57:16.374704 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:16.374683 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6bfcd89b85-9qb4s" Nov 03 19:57:16.400201 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:16.400139 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2xzzj" podStartSLOduration=2.063911932 podStartE2EDuration="3.400099505s" totalImagesPullingTime="1.336187573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:16.398885796 +0000 UTC m=+19.330753300" watchObservedRunningTime="2026-11-03 19:57:16.400099505 +0000 UTC m=+19.331967010" Nov 03 19:57:17.383413 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:17.383375 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k9x2f" event={"ID":"1fc8cec5-9248-4a47-822e-d81b37a380ef","Type":"ContainerStarted","Data":"fb99fa36ee7e66fe353352ba4f9543ec1d392fc12c7b9fc60c116bae8589f311"} Nov 03 19:57:19.326877 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:19.326842 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xsh7l" Nov 03 19:57:21.147288 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.147254 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-spdwv"] Nov 03 19:57:21.189362 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.189333 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.223582 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.223547 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4861f4b5-e2d1-474a-9f1b-dac138b37496-node-exporter-textfile\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.223732 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.223588 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4861f4b5-e2d1-474a-9f1b-dac138b37496-node-exporter-accelerators-collector-config\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.223732 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.223665 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4861f4b5-e2d1-474a-9f1b-dac138b37496-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.223732 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.223703 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4861f4b5-e2d1-474a-9f1b-dac138b37496-node-exporter-tls\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.223732 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.223727 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhqxn\" (UniqueName: \"kubernetes.io/projected/4861f4b5-e2d1-474a-9f1b-dac138b37496-kube-api-access-mhqxn\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.223901 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.223787 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4861f4b5-e2d1-474a-9f1b-dac138b37496-sys\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.223901 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.223805 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4861f4b5-e2d1-474a-9f1b-dac138b37496-metrics-client-ca\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.223901 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.223831 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4861f4b5-e2d1-474a-9f1b-dac138b37496-root\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.223901 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.223846 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4861f4b5-e2d1-474a-9f1b-dac138b37496-node-exporter-wtmp\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.246026 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.245997 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 19:57:21.246157 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.246079 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fksvw\"" Nov 03 19:57:21.246157 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.246099 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 19:57:21.256347 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.256322 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 19:57:21.256477 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.256459 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 19:57:21.324352 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.324328 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4861f4b5-e2d1-474a-9f1b-dac138b37496-node-exporter-tls\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.324511 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.324362 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhqxn\" (UniqueName: \"kubernetes.io/projected/4861f4b5-e2d1-474a-9f1b-dac138b37496-kube-api-access-mhqxn\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.324511 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.324391 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4861f4b5-e2d1-474a-9f1b-dac138b37496-sys\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.324511 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.324407 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4861f4b5-e2d1-474a-9f1b-dac138b37496-metrics-client-ca\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.324511 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.324433 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4861f4b5-e2d1-474a-9f1b-dac138b37496-root\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.324511 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.324472 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4861f4b5-e2d1-474a-9f1b-dac138b37496-root\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.324511 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.324477 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4861f4b5-e2d1-474a-9f1b-dac138b37496-sys\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.324511 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.324503 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4861f4b5-e2d1-474a-9f1b-dac138b37496-node-exporter-wtmp\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.324867 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.324531 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4861f4b5-e2d1-474a-9f1b-dac138b37496-node-exporter-textfile\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.324867 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.324550 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4861f4b5-e2d1-474a-9f1b-dac138b37496-node-exporter-accelerators-collector-config\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.324867 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.324580 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4861f4b5-e2d1-474a-9f1b-dac138b37496-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.324867 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.324690 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4861f4b5-e2d1-474a-9f1b-dac138b37496-node-exporter-wtmp\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.325071 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.324900 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4861f4b5-e2d1-474a-9f1b-dac138b37496-node-exporter-textfile\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.325204 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.325162 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4861f4b5-e2d1-474a-9f1b-dac138b37496-metrics-client-ca\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.325204 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.325193 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4861f4b5-e2d1-474a-9f1b-dac138b37496-node-exporter-accelerators-collector-config\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.326756 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.326739 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4861f4b5-e2d1-474a-9f1b-dac138b37496-node-exporter-tls\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.326812 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.326765 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4861f4b5-e2d1-474a-9f1b-dac138b37496-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.336202 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.336155 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhqxn\" (UniqueName: \"kubernetes.io/projected/4861f4b5-e2d1-474a-9f1b-dac138b37496-kube-api-access-mhqxn\") pod \"node-exporter-spdwv\" (UID: \"4861f4b5-e2d1-474a-9f1b-dac138b37496\") " pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.546062 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:21.545989 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-spdwv" Nov 03 19:57:21.586297 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:57:21.586269 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4861f4b5_e2d1_474a_9f1b_dac138b37496.slice/crio-cb286510342fd52bb69132bbf49fa10eb0b7f7997a550894223286741053880c WatchSource:0}: Error finding container cb286510342fd52bb69132bbf49fa10eb0b7f7997a550894223286741053880c: Status 404 returned error can't find the container with id cb286510342fd52bb69132bbf49fa10eb0b7f7997a550894223286741053880c Nov 03 19:57:22.405195 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:22.405152 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-spdwv" event={"ID":"4861f4b5-e2d1-474a-9f1b-dac138b37496","Type":"ContainerStarted","Data":"cb286510342fd52bb69132bbf49fa10eb0b7f7997a550894223286741053880c"} Nov 03 19:57:23.412410 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:23.412376 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-spdwv" event={"ID":"4861f4b5-e2d1-474a-9f1b-dac138b37496","Type":"ContainerStarted","Data":"2e4a6f81a79ecf6cb237c992ba9ce36f5cadbe5efa4642f391200f9d20a48048"} Nov 03 19:57:23.844498 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:23.844470 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:57:24.146130 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:24.146054 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:57:24.418272 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:24.418170 4918 generic.go:356] "Generic (PLEG): container finished" podID="4861f4b5-e2d1-474a-9f1b-dac138b37496" containerID="2e4a6f81a79ecf6cb237c992ba9ce36f5cadbe5efa4642f391200f9d20a48048" exitCode=0 Nov 03 19:57:24.418272 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:24.418217 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-spdwv" event={"ID":"4861f4b5-e2d1-474a-9f1b-dac138b37496","Type":"ContainerDied","Data":"2e4a6f81a79ecf6cb237c992ba9ce36f5cadbe5efa4642f391200f9d20a48048"} Nov 03 19:57:25.424640 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:25.424601 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-spdwv" event={"ID":"4861f4b5-e2d1-474a-9f1b-dac138b37496","Type":"ContainerStarted","Data":"ba0dbc33a4b3d0b269c981dedb21c5cef65f3f2a832059f43a2836ea42c7eb5e"} Nov 03 19:57:25.425049 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:25.424657 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-spdwv" event={"ID":"4861f4b5-e2d1-474a-9f1b-dac138b37496","Type":"ContainerStarted","Data":"4a7f9a827cc5692368ab72ab46a6b595a6e8fa8b0c178fd0d54b30640cdcc673"} Nov 03 19:57:25.448800 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:25.448749 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-spdwv" podStartSLOduration=2.735272522 podStartE2EDuration="4.448737553s" totalImagesPullingTime="1.641472299s" totalInitContainerRuntime="71.992732ms" isStatefulPod=true podCreationTimestamp="2026-11-03 19:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:25.446842233 +0000 UTC m=+28.378709745" watchObservedRunningTime="2026-11-03 19:57:25.448737553 +0000 UTC m=+28.380605057" Nov 03 19:57:25.979703 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:25.979671 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zn2ms" Nov 03 19:57:28.207965 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:28.207927 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 19:57:35.913806 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:35.913770 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d95b6bf4b-dd7kz"] Nov 03 19:57:38.861971 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:38.861864 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" podUID="adc4cee9-00f0-4fac-b655-f58c21377362" containerName="registry" containerID="cri-o://b4a6efe2c227375e934add2ede1e00719ad2d2752368f2ae340ac5d897c4259b" gracePeriod=30 Nov 03 19:57:39.101107 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.101084 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:57:39.167720 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.167636 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adc4cee9-00f0-4fac-b655-f58c21377362-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adc4cee9-00f0-4fac-b655-f58c21377362-trusted-ca\") pod \"adc4cee9-00f0-4fac-b655-f58c21377362\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " Nov 03 19:57:39.167720 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.167696 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adc4cee9-00f0-4fac-b655-f58c21377362-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/adc4cee9-00f0-4fac-b655-f58c21377362-ca-trust-extracted\") pod \"adc4cee9-00f0-4fac-b655-f58c21377362\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " Nov 03 19:57:39.167948 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.167742 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls\") pod \"adc4cee9-00f0-4fac-b655-f58c21377362\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " Nov 03 19:57:39.167948 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.167802 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-kube-api-access-6sx9r\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-kube-api-access-6sx9r\") pod \"adc4cee9-00f0-4fac-b655-f58c21377362\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " Nov 03 19:57:39.167948 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.167874 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adc4cee9-00f0-4fac-b655-f58c21377362-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/adc4cee9-00f0-4fac-b655-f58c21377362-image-registry-private-configuration\") pod \"adc4cee9-00f0-4fac-b655-f58c21377362\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " Nov 03 19:57:39.168097 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.167949 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adc4cee9-00f0-4fac-b655-f58c21377362-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/adc4cee9-00f0-4fac-b655-f58c21377362-installation-pull-secrets\") pod \"adc4cee9-00f0-4fac-b655-f58c21377362\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " Nov 03 19:57:39.168097 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.167987 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adc4cee9-00f0-4fac-b655-f58c21377362-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/adc4cee9-00f0-4fac-b655-f58c21377362-registry-certificates\") pod \"adc4cee9-00f0-4fac-b655-f58c21377362\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " Nov 03 19:57:39.168097 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.168025 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-bound-sa-token\") pod \"adc4cee9-00f0-4fac-b655-f58c21377362\" (UID: \"adc4cee9-00f0-4fac-b655-f58c21377362\") " Nov 03 19:57:39.168296 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.168129 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc4cee9-00f0-4fac-b655-f58c21377362-trusted-ca" pod "adc4cee9-00f0-4fac-b655-f58c21377362" (UID: "adc4cee9-00f0-4fac-b655-f58c21377362"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:57:39.169068 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.169011 4918 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adc4cee9-00f0-4fac-b655-f58c21377362-trusted-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:39.170764 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.170663 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-kube-api-access-6sx9r" pod "adc4cee9-00f0-4fac-b655-f58c21377362" (UID: "adc4cee9-00f0-4fac-b655-f58c21377362"). InnerVolumeSpecName "kube-api-access-6sx9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:57:39.171084 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.170964 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc4cee9-00f0-4fac-b655-f58c21377362-image-registry-private-configuration" pod "adc4cee9-00f0-4fac-b655-f58c21377362" (UID: "adc4cee9-00f0-4fac-b655-f58c21377362"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:57:39.171084 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.170998 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-bound-sa-token" pod "adc4cee9-00f0-4fac-b655-f58c21377362" (UID: "adc4cee9-00f0-4fac-b655-f58c21377362"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:57:39.171261 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.171129 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls" pod "adc4cee9-00f0-4fac-b655-f58c21377362" (UID: "adc4cee9-00f0-4fac-b655-f58c21377362"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:57:39.171776 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.171435 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc4cee9-00f0-4fac-b655-f58c21377362-registry-certificates" pod "adc4cee9-00f0-4fac-b655-f58c21377362" (UID: "adc4cee9-00f0-4fac-b655-f58c21377362"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:57:39.175472 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.175369 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc4cee9-00f0-4fac-b655-f58c21377362-installation-pull-secrets" pod "adc4cee9-00f0-4fac-b655-f58c21377362" (UID: "adc4cee9-00f0-4fac-b655-f58c21377362"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:57:39.177347 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.177315 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adc4cee9-00f0-4fac-b655-f58c21377362-ca-trust-extracted" pod "adc4cee9-00f0-4fac-b655-f58c21377362" (UID: "adc4cee9-00f0-4fac-b655-f58c21377362"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:39.270229 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.270193 4918 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/adc4cee9-00f0-4fac-b655-f58c21377362-installation-pull-secrets\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:39.270229 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.270224 4918 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/adc4cee9-00f0-4fac-b655-f58c21377362-registry-certificates\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:39.270229 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.270236 4918 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-bound-sa-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:39.270474 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.270245 4918 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/adc4cee9-00f0-4fac-b655-f58c21377362-ca-trust-extracted\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:39.270474 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.270256 4918 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-registry-tls\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:39.270474 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.270264 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sx9r\" (UniqueName: \"kubernetes.io/projected/adc4cee9-00f0-4fac-b655-f58c21377362-kube-api-access-6sx9r\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:39.270474 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.270273 4918 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/adc4cee9-00f0-4fac-b655-f58c21377362-image-registry-private-configuration\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:39.481298 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.481215 4918 generic.go:356] "Generic (PLEG): container finished" podID="adc4cee9-00f0-4fac-b655-f58c21377362" containerID="b4a6efe2c227375e934add2ede1e00719ad2d2752368f2ae340ac5d897c4259b" exitCode=0 Nov 03 19:57:39.481298 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.481276 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" Nov 03 19:57:39.481470 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.481300 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" event={"ID":"adc4cee9-00f0-4fac-b655-f58c21377362","Type":"ContainerDied","Data":"b4a6efe2c227375e934add2ede1e00719ad2d2752368f2ae340ac5d897c4259b"} Nov 03 19:57:39.481470 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.481354 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9d99c5c47-ttssq" event={"ID":"adc4cee9-00f0-4fac-b655-f58c21377362","Type":"ContainerDied","Data":"8327382533dc4728e498c02f706cd469101a79ec1aaa2579372bbb24e180b893"} Nov 03 19:57:39.481470 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.481375 4918 scope.go:122] "RemoveContainer" containerID="b4a6efe2c227375e934add2ede1e00719ad2d2752368f2ae340ac5d897c4259b" Nov 03 19:57:39.489450 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.489435 4918 scope.go:122] "RemoveContainer" containerID="b4a6efe2c227375e934add2ede1e00719ad2d2752368f2ae340ac5d897c4259b" Nov 03 19:57:39.489719 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:39.489700 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4a6efe2c227375e934add2ede1e00719ad2d2752368f2ae340ac5d897c4259b\": container with ID starting with b4a6efe2c227375e934add2ede1e00719ad2d2752368f2ae340ac5d897c4259b not found: ID does not exist" containerID="b4a6efe2c227375e934add2ede1e00719ad2d2752368f2ae340ac5d897c4259b" Nov 03 19:57:39.489770 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.489727 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a6efe2c227375e934add2ede1e00719ad2d2752368f2ae340ac5d897c4259b"} err="failed to get container status \"b4a6efe2c227375e934add2ede1e00719ad2d2752368f2ae340ac5d897c4259b\": rpc error: code = NotFound desc = could not find container \"b4a6efe2c227375e934add2ede1e00719ad2d2752368f2ae340ac5d897c4259b\": container with ID starting with b4a6efe2c227375e934add2ede1e00719ad2d2752368f2ae340ac5d897c4259b not found: ID does not exist" Nov 03 19:57:39.498602 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.498582 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9d99c5c47-ttssq"] Nov 03 19:57:39.502364 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:39.502344 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-9d99c5c47-ttssq"] Nov 03 19:57:41.212693 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:57:41.212656 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adc4cee9-00f0-4fac-b655-f58c21377362" path="/var/lib/kubelet/pods/adc4cee9-00f0-4fac-b655-f58c21377362/volumes" Nov 03 19:57:58.206816 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:57:58.206777 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 19:58:00.933685 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:00.933625 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" podUID="514e4bfb-77ef-4b1f-977a-3f9308cf886c" containerName="registry" containerID="cri-o://bcd99734f750dfcfafa967e0a20102f5771824b4430adbbbc5559882a43b7ab3" gracePeriod=30 Nov 03 19:58:01.178793 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.178770 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:58:01.362160 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.362127 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-bound-sa-token\") pod \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " Nov 03 19:58:01.362378 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.362167 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/514e4bfb-77ef-4b1f-977a-3f9308cf886c-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/514e4bfb-77ef-4b1f-977a-3f9308cf886c-installation-pull-secrets\") pod \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " Nov 03 19:58:01.362378 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.362221 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-kube-api-access-djg6c\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-kube-api-access-djg6c\") pod \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " Nov 03 19:58:01.362378 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.362292 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/514e4bfb-77ef-4b1f-977a-3f9308cf886c-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/514e4bfb-77ef-4b1f-977a-3f9308cf886c-image-registry-private-configuration\") pod \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " Nov 03 19:58:01.362378 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.362321 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/514e4bfb-77ef-4b1f-977a-3f9308cf886c-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/514e4bfb-77ef-4b1f-977a-3f9308cf886c-ca-trust-extracted\") pod \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " Nov 03 19:58:01.362378 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.362340 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls\") pod \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " Nov 03 19:58:01.362378 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.362374 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-certificates\") pod \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " Nov 03 19:58:01.362710 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.362403 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/514e4bfb-77ef-4b1f-977a-3f9308cf886c-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/514e4bfb-77ef-4b1f-977a-3f9308cf886c-trusted-ca\") pod \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\" (UID: \"514e4bfb-77ef-4b1f-977a-3f9308cf886c\") " Nov 03 19:58:01.363021 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.362994 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514e4bfb-77ef-4b1f-977a-3f9308cf886c-trusted-ca" pod "514e4bfb-77ef-4b1f-977a-3f9308cf886c" (UID: "514e4bfb-77ef-4b1f-977a-3f9308cf886c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:01.363521 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.363491 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-certificates" pod "514e4bfb-77ef-4b1f-977a-3f9308cf886c" (UID: "514e4bfb-77ef-4b1f-977a-3f9308cf886c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:01.364727 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.364697 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514e4bfb-77ef-4b1f-977a-3f9308cf886c-installation-pull-secrets" pod "514e4bfb-77ef-4b1f-977a-3f9308cf886c" (UID: "514e4bfb-77ef-4b1f-977a-3f9308cf886c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:01.364875 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.364794 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514e4bfb-77ef-4b1f-977a-3f9308cf886c-image-registry-private-configuration" pod "514e4bfb-77ef-4b1f-977a-3f9308cf886c" (UID: "514e4bfb-77ef-4b1f-977a-3f9308cf886c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:01.364997 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.364869 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-bound-sa-token" pod "514e4bfb-77ef-4b1f-977a-3f9308cf886c" (UID: "514e4bfb-77ef-4b1f-977a-3f9308cf886c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:01.365050 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.364986 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-kube-api-access-djg6c" pod "514e4bfb-77ef-4b1f-977a-3f9308cf886c" (UID: "514e4bfb-77ef-4b1f-977a-3f9308cf886c"). InnerVolumeSpecName "kube-api-access-djg6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:01.365109 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.365092 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls" pod "514e4bfb-77ef-4b1f-977a-3f9308cf886c" (UID: "514e4bfb-77ef-4b1f-977a-3f9308cf886c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:01.369426 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.369404 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/514e4bfb-77ef-4b1f-977a-3f9308cf886c-ca-trust-extracted" pod "514e4bfb-77ef-4b1f-977a-3f9308cf886c" (UID: "514e4bfb-77ef-4b1f-977a-3f9308cf886c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:01.464026 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.463985 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djg6c\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-kube-api-access-djg6c\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:01.464026 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.464018 4918 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/514e4bfb-77ef-4b1f-977a-3f9308cf886c-image-registry-private-configuration\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:01.464026 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.464034 4918 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/514e4bfb-77ef-4b1f-977a-3f9308cf886c-ca-trust-extracted\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:01.464371 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.464046 4918 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-tls\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:01.464371 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.464055 4918 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/514e4bfb-77ef-4b1f-977a-3f9308cf886c-registry-certificates\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:01.464371 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.464065 4918 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/514e4bfb-77ef-4b1f-977a-3f9308cf886c-trusted-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:01.464371 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.464074 4918 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/514e4bfb-77ef-4b1f-977a-3f9308cf886c-bound-sa-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:01.464371 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.464082 4918 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/514e4bfb-77ef-4b1f-977a-3f9308cf886c-installation-pull-secrets\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:01.574927 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.574884 4918 generic.go:356] "Generic (PLEG): container finished" podID="514e4bfb-77ef-4b1f-977a-3f9308cf886c" containerID="bcd99734f750dfcfafa967e0a20102f5771824b4430adbbbc5559882a43b7ab3" exitCode=0 Nov 03 19:58:01.575101 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.574948 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" Nov 03 19:58:01.575101 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.574980 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" event={"ID":"514e4bfb-77ef-4b1f-977a-3f9308cf886c","Type":"ContainerDied","Data":"bcd99734f750dfcfafa967e0a20102f5771824b4430adbbbc5559882a43b7ab3"} Nov 03 19:58:01.575101 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.575050 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d95b6bf4b-dd7kz" event={"ID":"514e4bfb-77ef-4b1f-977a-3f9308cf886c","Type":"ContainerDied","Data":"d9837cebc843f195a9d4d3c8f65191fef8d20657f1a8ec367dcb5f4783de7716"} Nov 03 19:58:01.575101 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.575077 4918 scope.go:122] "RemoveContainer" containerID="bcd99734f750dfcfafa967e0a20102f5771824b4430adbbbc5559882a43b7ab3" Nov 03 19:58:01.583442 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.583425 4918 scope.go:122] "RemoveContainer" containerID="bcd99734f750dfcfafa967e0a20102f5771824b4430adbbbc5559882a43b7ab3" Nov 03 19:58:01.583712 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:58:01.583694 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcd99734f750dfcfafa967e0a20102f5771824b4430adbbbc5559882a43b7ab3\": container with ID starting with bcd99734f750dfcfafa967e0a20102f5771824b4430adbbbc5559882a43b7ab3 not found: ID does not exist" containerID="bcd99734f750dfcfafa967e0a20102f5771824b4430adbbbc5559882a43b7ab3" Nov 03 19:58:01.583755 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.583722 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcd99734f750dfcfafa967e0a20102f5771824b4430adbbbc5559882a43b7ab3"} err="failed to get container status \"bcd99734f750dfcfafa967e0a20102f5771824b4430adbbbc5559882a43b7ab3\": rpc error: code = NotFound desc = could not find container \"bcd99734f750dfcfafa967e0a20102f5771824b4430adbbbc5559882a43b7ab3\": container with ID starting with bcd99734f750dfcfafa967e0a20102f5771824b4430adbbbc5559882a43b7ab3 not found: ID does not exist" Nov 03 19:58:01.596148 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.596118 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d95b6bf4b-dd7kz"] Nov 03 19:58:01.603468 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:01.603448 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d95b6bf4b-dd7kz"] Nov 03 19:58:03.212630 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:03.212599 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="514e4bfb-77ef-4b1f-977a-3f9308cf886c" path="/var/lib/kubelet/pods/514e4bfb-77ef-4b1f-977a-3f9308cf886c/volumes" Nov 03 19:58:07.598789 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:07.598755 4918 generic.go:356] "Generic (PLEG): container finished" podID="ff051e4a-95a6-4c0d-9f69-773974d6eb7c" containerID="bc41be73cadc57028af935a02e8905446331f52ffec67f21722dcc527e212f07" exitCode=0 Nov 03 19:58:07.599298 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:07.598833 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" event={"ID":"ff051e4a-95a6-4c0d-9f69-773974d6eb7c","Type":"ContainerDied","Data":"bc41be73cadc57028af935a02e8905446331f52ffec67f21722dcc527e212f07"} Nov 03 19:58:07.599298 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:07.599229 4918 scope.go:122] "RemoveContainer" containerID="bc41be73cadc57028af935a02e8905446331f52ffec67f21722dcc527e212f07" Nov 03 19:58:07.600378 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:07.600357 4918 generic.go:356] "Generic (PLEG): container finished" podID="449a05ee-ead2-4507-8d65-faaeb3ab3281" containerID="f692effef71642f90293457632bf5f1c98c1b52a85d830c4979728f184683093" exitCode=0 Nov 03 19:58:07.600459 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:07.600419 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zl889" event={"ID":"449a05ee-ead2-4507-8d65-faaeb3ab3281","Type":"ContainerDied","Data":"f692effef71642f90293457632bf5f1c98c1b52a85d830c4979728f184683093"} Nov 03 19:58:07.600694 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:07.600682 4918 scope.go:122] "RemoveContainer" containerID="f692effef71642f90293457632bf5f1c98c1b52a85d830c4979728f184683093" Nov 03 19:58:07.973230 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:07.973134 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-l984r-mntts"] Nov 03 19:58:07.973463 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:07.973435 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-l984r-mntts" podUID="33b6188c-0202-4bc8-813f-2d189587f3aa" containerName="insights-gathering" containerID="cri-o://4dea61c608711b32be419f6e4b8734e863f0ca5346d7ad77075c0833a1d33de8" gracePeriod=30 Nov 03 19:58:08.071630 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.071597 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-rt7km-2v6j4"] Nov 03 19:58:08.071912 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.071898 4918 cpu_manager.go:414] "containerMap: removing container" podUID="adc4cee9-00f0-4fac-b655-f58c21377362" containerName="registry" Nov 03 19:58:08.071962 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.071913 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adc4cee9-00f0-4fac-b655-f58c21377362" containerName="registry" Nov 03 19:58:08.071962 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.071939 4918 cpu_manager.go:414] "containerMap: removing container" podUID="514e4bfb-77ef-4b1f-977a-3f9308cf886c" containerName="registry" Nov 03 19:58:08.071962 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.071945 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="514e4bfb-77ef-4b1f-977a-3f9308cf886c" containerName="registry" Nov 03 19:58:08.072060 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.071990 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="adc4cee9-00f0-4fac-b655-f58c21377362" containerName="registry" Nov 03 19:58:08.072060 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.072001 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="514e4bfb-77ef-4b1f-977a-3f9308cf886c" containerName="registry" Nov 03 19:58:08.075221 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.075204 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rt7km-2v6j4" Nov 03 19:58:08.084008 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.083980 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rt7km-2v6j4"] Nov 03 19:58:08.213961 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.213931 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svwkt\" (UniqueName: \"kubernetes.io/projected/94b59eff-2de3-4954-8905-3d63490df350-kube-api-access-svwkt\") pod \"periodic-gathering-rt7km-2v6j4\" (UID: \"94b59eff-2de3-4954-8905-3d63490df350\") " pod="openshift-insights/periodic-gathering-rt7km-2v6j4" Nov 03 19:58:08.214078 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.213968 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/94b59eff-2de3-4954-8905-3d63490df350-archives-path\") pod \"periodic-gathering-rt7km-2v6j4\" (UID: \"94b59eff-2de3-4954-8905-3d63490df350\") " pod="openshift-insights/periodic-gathering-rt7km-2v6j4" Nov 03 19:58:08.214078 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.213991 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b59eff-2de3-4954-8905-3d63490df350-service-ca-bundle\") pod \"periodic-gathering-rt7km-2v6j4\" (UID: \"94b59eff-2de3-4954-8905-3d63490df350\") " pod="openshift-insights/periodic-gathering-rt7km-2v6j4" Nov 03 19:58:08.214208 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.214170 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b59eff-2de3-4954-8905-3d63490df350-trusted-ca-bundle\") pod \"periodic-gathering-rt7km-2v6j4\" (UID: \"94b59eff-2de3-4954-8905-3d63490df350\") " pod="openshift-insights/periodic-gathering-rt7km-2v6j4" Nov 03 19:58:08.218645 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.218624 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:58:08.315251 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.315222 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b6188c-0202-4bc8-813f-2d189587f3aa-kube-api-access-gxblp\" (UniqueName: \"kubernetes.io/projected/33b6188c-0202-4bc8-813f-2d189587f3aa-kube-api-access-gxblp\") pod \"33b6188c-0202-4bc8-813f-2d189587f3aa\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " Nov 03 19:58:08.315411 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.315266 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b6188c-0202-4bc8-813f-2d189587f3aa-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/33b6188c-0202-4bc8-813f-2d189587f3aa-archives-path\") pod \"33b6188c-0202-4bc8-813f-2d189587f3aa\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " Nov 03 19:58:08.315411 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.315295 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b6188c-0202-4bc8-813f-2d189587f3aa-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33b6188c-0202-4bc8-813f-2d189587f3aa-trusted-ca-bundle\") pod \"33b6188c-0202-4bc8-813f-2d189587f3aa\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " Nov 03 19:58:08.315411 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.315319 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b6188c-0202-4bc8-813f-2d189587f3aa-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33b6188c-0202-4bc8-813f-2d189587f3aa-service-ca-bundle\") pod \"33b6188c-0202-4bc8-813f-2d189587f3aa\" (UID: \"33b6188c-0202-4bc8-813f-2d189587f3aa\") " Nov 03 19:58:08.315560 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.315420 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/94b59eff-2de3-4954-8905-3d63490df350-archives-path\") pod \"periodic-gathering-rt7km-2v6j4\" (UID: \"94b59eff-2de3-4954-8905-3d63490df350\") " pod="openshift-insights/periodic-gathering-rt7km-2v6j4" Nov 03 19:58:08.315560 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.315447 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b59eff-2de3-4954-8905-3d63490df350-service-ca-bundle\") pod \"periodic-gathering-rt7km-2v6j4\" (UID: \"94b59eff-2de3-4954-8905-3d63490df350\") " pod="openshift-insights/periodic-gathering-rt7km-2v6j4" Nov 03 19:58:08.315560 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.315511 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b59eff-2de3-4954-8905-3d63490df350-trusted-ca-bundle\") pod \"periodic-gathering-rt7km-2v6j4\" (UID: \"94b59eff-2de3-4954-8905-3d63490df350\") " pod="openshift-insights/periodic-gathering-rt7km-2v6j4" Nov 03 19:58:08.315560 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.315543 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svwkt\" (UniqueName: \"kubernetes.io/projected/94b59eff-2de3-4954-8905-3d63490df350-kube-api-access-svwkt\") pod \"periodic-gathering-rt7km-2v6j4\" (UID: \"94b59eff-2de3-4954-8905-3d63490df350\") " pod="openshift-insights/periodic-gathering-rt7km-2v6j4" Nov 03 19:58:08.315767 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.315682 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b6188c-0202-4bc8-813f-2d189587f3aa-archives-path" pod "33b6188c-0202-4bc8-813f-2d189587f3aa" (UID: "33b6188c-0202-4bc8-813f-2d189587f3aa"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:08.315767 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.315750 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b6188c-0202-4bc8-813f-2d189587f3aa-trusted-ca-bundle" pod "33b6188c-0202-4bc8-813f-2d189587f3aa" (UID: "33b6188c-0202-4bc8-813f-2d189587f3aa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:08.315859 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.315780 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b6188c-0202-4bc8-813f-2d189587f3aa-service-ca-bundle" pod "33b6188c-0202-4bc8-813f-2d189587f3aa" (UID: "33b6188c-0202-4bc8-813f-2d189587f3aa"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:08.315898 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.315879 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/94b59eff-2de3-4954-8905-3d63490df350-archives-path\") pod \"periodic-gathering-rt7km-2v6j4\" (UID: \"94b59eff-2de3-4954-8905-3d63490df350\") " pod="openshift-insights/periodic-gathering-rt7km-2v6j4" Nov 03 19:58:08.316330 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.316311 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b59eff-2de3-4954-8905-3d63490df350-service-ca-bundle\") pod \"periodic-gathering-rt7km-2v6j4\" (UID: \"94b59eff-2de3-4954-8905-3d63490df350\") " pod="openshift-insights/periodic-gathering-rt7km-2v6j4" Nov 03 19:58:08.316614 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.316593 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b59eff-2de3-4954-8905-3d63490df350-trusted-ca-bundle\") pod \"periodic-gathering-rt7km-2v6j4\" (UID: \"94b59eff-2de3-4954-8905-3d63490df350\") " pod="openshift-insights/periodic-gathering-rt7km-2v6j4" Nov 03 19:58:08.317660 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.317624 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b6188c-0202-4bc8-813f-2d189587f3aa-kube-api-access-gxblp" pod "33b6188c-0202-4bc8-813f-2d189587f3aa" (UID: "33b6188c-0202-4bc8-813f-2d189587f3aa"). InnerVolumeSpecName "kube-api-access-gxblp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:08.328153 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.328130 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svwkt\" (UniqueName: \"kubernetes.io/projected/94b59eff-2de3-4954-8905-3d63490df350-kube-api-access-svwkt\") pod \"periodic-gathering-rt7km-2v6j4\" (UID: \"94b59eff-2de3-4954-8905-3d63490df350\") " pod="openshift-insights/periodic-gathering-rt7km-2v6j4" Nov 03 19:58:08.388010 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.387963 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rt7km-2v6j4" Nov 03 19:58:08.416543 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.416518 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxblp\" (UniqueName: \"kubernetes.io/projected/33b6188c-0202-4bc8-813f-2d189587f3aa-kube-api-access-gxblp\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:08.416543 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.416544 4918 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/33b6188c-0202-4bc8-813f-2d189587f3aa-archives-path\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:08.416690 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.416556 4918 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33b6188c-0202-4bc8-813f-2d189587f3aa-trusted-ca-bundle\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:08.416690 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.416567 4918 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33b6188c-0202-4bc8-813f-2d189587f3aa-service-ca-bundle\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:08.605834 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.605747 4918 generic.go:356] "Generic (PLEG): container finished" podID="33b6188c-0202-4bc8-813f-2d189587f3aa" containerID="4dea61c608711b32be419f6e4b8734e863f0ca5346d7ad77075c0833a1d33de8" exitCode=2 Nov 03 19:58:08.605834 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.605795 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-l984r-mntts" Nov 03 19:58:08.606311 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.605842 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-l984r-mntts" event={"ID":"33b6188c-0202-4bc8-813f-2d189587f3aa","Type":"ContainerDied","Data":"4dea61c608711b32be419f6e4b8734e863f0ca5346d7ad77075c0833a1d33de8"} Nov 03 19:58:08.606311 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.605891 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-l984r-mntts" event={"ID":"33b6188c-0202-4bc8-813f-2d189587f3aa","Type":"ContainerDied","Data":"1ca9416f4f2401864659a5a0f7debfb2d9b3f44c8f9b1ff02a8fa40dca952bb5"} Nov 03 19:58:08.606311 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.605917 4918 scope.go:122] "RemoveContainer" containerID="4dea61c608711b32be419f6e4b8734e863f0ca5346d7ad77075c0833a1d33de8" Nov 03 19:58:08.607797 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.607777 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sc7wq" event={"ID":"ff051e4a-95a6-4c0d-9f69-773974d6eb7c","Type":"ContainerStarted","Data":"a046aa2cd777b7e8ff016b0b0ce2343e3127b309fe7a2c222f4e6671b3faff0a"} Nov 03 19:58:08.609712 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.609690 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zl889" event={"ID":"449a05ee-ead2-4507-8d65-faaeb3ab3281","Type":"ContainerStarted","Data":"887b07198adca437c4e4a5b84b746ad63424a725219a7e835ef32a9989be298b"} Nov 03 19:58:08.616226 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.616208 4918 scope.go:122] "RemoveContainer" containerID="4dea61c608711b32be419f6e4b8734e863f0ca5346d7ad77075c0833a1d33de8" Nov 03 19:58:08.616536 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:58:08.616512 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dea61c608711b32be419f6e4b8734e863f0ca5346d7ad77075c0833a1d33de8\": container with ID starting with 4dea61c608711b32be419f6e4b8734e863f0ca5346d7ad77075c0833a1d33de8 not found: ID does not exist" containerID="4dea61c608711b32be419f6e4b8734e863f0ca5346d7ad77075c0833a1d33de8" Nov 03 19:58:08.616616 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.616545 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dea61c608711b32be419f6e4b8734e863f0ca5346d7ad77075c0833a1d33de8"} err="failed to get container status \"4dea61c608711b32be419f6e4b8734e863f0ca5346d7ad77075c0833a1d33de8\": rpc error: code = NotFound desc = could not find container \"4dea61c608711b32be419f6e4b8734e863f0ca5346d7ad77075c0833a1d33de8\": container with ID starting with 4dea61c608711b32be419f6e4b8734e863f0ca5346d7ad77075c0833a1d33de8 not found: ID does not exist" Nov 03 19:58:08.663325 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.663296 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rt7km-2v6j4"] Nov 03 19:58:08.666984 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.666951 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-l984r-mntts"] Nov 03 19:58:08.667382 ip-10-0-139-53 kubenswrapper[4918]: W1103 19:58:08.667360 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b59eff_2de3_4954_8905_3d63490df350.slice/crio-41c4afb272ae3cfe730fc0b70bbf0bb10c78915ba6375b29e7f0f0988b486d88 WatchSource:0}: Error finding container 41c4afb272ae3cfe730fc0b70bbf0bb10c78915ba6375b29e7f0f0988b486d88: Status 404 returned error can't find the container with id 41c4afb272ae3cfe730fc0b70bbf0bb10c78915ba6375b29e7f0f0988b486d88 Nov 03 19:58:08.675120 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:08.675096 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-l984r-mntts"] Nov 03 19:58:09.212073 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:09.212028 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b6188c-0202-4bc8-813f-2d189587f3aa" path="/var/lib/kubelet/pods/33b6188c-0202-4bc8-813f-2d189587f3aa/volumes" Nov 03 19:58:09.616466 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:09.616435 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rt7km-2v6j4" event={"ID":"94b59eff-2de3-4954-8905-3d63490df350","Type":"ContainerStarted","Data":"6be1627fd427e5e03cf0f10415ef784642c3e4c02b1bc2502d5c9b0d8f800fa1"} Nov 03 19:58:09.616833 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:09.616483 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rt7km-2v6j4" event={"ID":"94b59eff-2de3-4954-8905-3d63490df350","Type":"ContainerStarted","Data":"41c4afb272ae3cfe730fc0b70bbf0bb10c78915ba6375b29e7f0f0988b486d88"} Nov 03 19:58:09.635777 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:09.635731 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-rt7km-2v6j4" podStartSLOduration=1.635718131 podStartE2EDuration="1.635718131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:09.634115769 +0000 UTC m=+72.565983270" watchObservedRunningTime="2026-11-03 19:58:09.635718131 +0000 UTC m=+72.567585644" Nov 03 19:58:12.632982 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:12.632948 4918 generic.go:356] "Generic (PLEG): container finished" podID="23f64ae9-fdda-4f2e-b3a5-1c552cc55a44" containerID="645219b66f36e5a9367977b6bc1310035e4ff45f0aea585089a4b49ce24a1e0d" exitCode=0 Nov 03 19:58:12.633421 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:12.633024 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" event={"ID":"23f64ae9-fdda-4f2e-b3a5-1c552cc55a44","Type":"ContainerDied","Data":"645219b66f36e5a9367977b6bc1310035e4ff45f0aea585089a4b49ce24a1e0d"} Nov 03 19:58:12.633421 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:12.633369 4918 scope.go:122] "RemoveContainer" containerID="645219b66f36e5a9367977b6bc1310035e4ff45f0aea585089a4b49ce24a1e0d" Nov 03 19:58:13.639404 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:58:13.639372 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ncj4" event={"ID":"23f64ae9-fdda-4f2e-b3a5-1c552cc55a44","Type":"ContainerStarted","Data":"274981dfc24ba133bce666e36c9516edec3c750c70275bf8302450b7e2b89eec"} Nov 03 19:58:28.216891 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:58:28.216854 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 19:58:58.206610 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:58:58.206572 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 19:59:28.206046 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:59:28.205997 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 19:59:31.929231 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:59:31.929199 4918 generic.go:356] "Generic (PLEG): container finished" podID="94b59eff-2de3-4954-8905-3d63490df350" containerID="6be1627fd427e5e03cf0f10415ef784642c3e4c02b1bc2502d5c9b0d8f800fa1" exitCode=0 Nov 03 19:59:31.929634 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:59:31.929255 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rt7km-2v6j4" event={"ID":"94b59eff-2de3-4954-8905-3d63490df350","Type":"ContainerDied","Data":"6be1627fd427e5e03cf0f10415ef784642c3e4c02b1bc2502d5c9b0d8f800fa1"} Nov 03 19:59:33.064216 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:59:33.064170 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rt7km-2v6j4" Nov 03 19:59:33.194033 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:59:33.193935 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94b59eff-2de3-4954-8905-3d63490df350-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b59eff-2de3-4954-8905-3d63490df350-service-ca-bundle\") pod \"94b59eff-2de3-4954-8905-3d63490df350\" (UID: \"94b59eff-2de3-4954-8905-3d63490df350\") " Nov 03 19:59:33.194033 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:59:33.193985 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b59eff-2de3-4954-8905-3d63490df350-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/94b59eff-2de3-4954-8905-3d63490df350-archives-path\") pod \"94b59eff-2de3-4954-8905-3d63490df350\" (UID: \"94b59eff-2de3-4954-8905-3d63490df350\") " Nov 03 19:59:33.194298 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:59:33.194045 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b59eff-2de3-4954-8905-3d63490df350-kube-api-access-svwkt\" (UniqueName: \"kubernetes.io/projected/94b59eff-2de3-4954-8905-3d63490df350-kube-api-access-svwkt\") pod \"94b59eff-2de3-4954-8905-3d63490df350\" (UID: \"94b59eff-2de3-4954-8905-3d63490df350\") " Nov 03 19:59:33.194298 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:59:33.194082 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94b59eff-2de3-4954-8905-3d63490df350-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b59eff-2de3-4954-8905-3d63490df350-trusted-ca-bundle\") pod \"94b59eff-2de3-4954-8905-3d63490df350\" (UID: \"94b59eff-2de3-4954-8905-3d63490df350\") " Nov 03 19:59:33.194410 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:59:33.194387 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b59eff-2de3-4954-8905-3d63490df350-archives-path" pod "94b59eff-2de3-4954-8905-3d63490df350" (UID: "94b59eff-2de3-4954-8905-3d63490df350"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:59:33.194461 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:59:33.194420 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b59eff-2de3-4954-8905-3d63490df350-service-ca-bundle" pod "94b59eff-2de3-4954-8905-3d63490df350" (UID: "94b59eff-2de3-4954-8905-3d63490df350"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:59:33.194503 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:59:33.194461 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b59eff-2de3-4954-8905-3d63490df350-trusted-ca-bundle" pod "94b59eff-2de3-4954-8905-3d63490df350" (UID: "94b59eff-2de3-4954-8905-3d63490df350"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:59:33.196072 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:59:33.196051 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b59eff-2de3-4954-8905-3d63490df350-kube-api-access-svwkt" pod "94b59eff-2de3-4954-8905-3d63490df350" (UID: "94b59eff-2de3-4954-8905-3d63490df350"). InnerVolumeSpecName "kube-api-access-svwkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:59:33.295164 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:59:33.295126 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svwkt\" (UniqueName: \"kubernetes.io/projected/94b59eff-2de3-4954-8905-3d63490df350-kube-api-access-svwkt\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:33.295164 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:59:33.295159 4918 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b59eff-2de3-4954-8905-3d63490df350-trusted-ca-bundle\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:33.295380 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:59:33.295197 4918 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b59eff-2de3-4954-8905-3d63490df350-service-ca-bundle\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:33.295380 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:59:33.295211 4918 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/94b59eff-2de3-4954-8905-3d63490df350-archives-path\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:33.937953 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:59:33.937925 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rt7km-2v6j4" Nov 03 19:59:33.938126 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:59:33.937919 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rt7km-2v6j4" event={"ID":"94b59eff-2de3-4954-8905-3d63490df350","Type":"ContainerDied","Data":"41c4afb272ae3cfe730fc0b70bbf0bb10c78915ba6375b29e7f0f0988b486d88"} Nov 03 19:59:33.938126 ip-10-0-139-53 kubenswrapper[4918]: I1103 19:59:33.938047 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41c4afb272ae3cfe730fc0b70bbf0bb10c78915ba6375b29e7f0f0988b486d88" Nov 03 19:59:58.206711 ip-10-0-139-53 kubenswrapper[4918]: E1103 19:59:58.206663 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:00:28.206508 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:00:28.206470 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:00:58.206508 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:00:58.206468 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:01:28.206380 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:01:28.206336 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:01:57.227359 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:01:57.227332 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:01:57.227843 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:01:57.227824 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:01:58.206983 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:01:58.206936 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:02:28.206981 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:02:28.206941 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:02:58.206822 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:02:58.206776 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:03:19.064493 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.064452 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js"] Nov 03 20:03:19.064958 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.064762 4918 cpu_manager.go:414] "containerMap: removing container" podUID="94b59eff-2de3-4954-8905-3d63490df350" containerName="insights-gathering" Nov 03 20:03:19.064958 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.064773 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b59eff-2de3-4954-8905-3d63490df350" containerName="insights-gathering" Nov 03 20:03:19.064958 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.064792 4918 cpu_manager.go:414] "containerMap: removing container" podUID="33b6188c-0202-4bc8-813f-2d189587f3aa" containerName="insights-gathering" Nov 03 20:03:19.064958 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.064798 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b6188c-0202-4bc8-813f-2d189587f3aa" containerName="insights-gathering" Nov 03 20:03:19.064958 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.064855 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b6188c-0202-4bc8-813f-2d189587f3aa" containerName="insights-gathering" Nov 03 20:03:19.064958 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.064866 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="94b59eff-2de3-4954-8905-3d63490df350" containerName="insights-gathering" Nov 03 20:03:19.067773 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.067754 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.071602 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.071579 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 20:03:19.071755 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.071733 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 20:03:19.071868 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.071779 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 20:03:19.071868 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.071778 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 20:03:19.072466 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.072451 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xj9dq\"" Nov 03 20:03:19.083645 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.083620 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js"] Nov 03 20:03:19.171773 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.171729 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.171773 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.171770 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.171977 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.171794 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.171977 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.171914 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dfe3317-eeae-4bf2-a116-a152249b4f59-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.171977 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.171938 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.171977 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.171965 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65m87\" (UniqueName: \"kubernetes.io/projected/9dfe3317-eeae-4bf2-a116-a152249b4f59-kube-api-access-65m87\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.172103 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.171984 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.273137 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.273101 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.273297 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.273165 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dfe3317-eeae-4bf2-a116-a152249b4f59-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.273355 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.273309 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.273950 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.273543 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65m87\" (UniqueName: \"kubernetes.io/projected/9dfe3317-eeae-4bf2-a116-a152249b4f59-kube-api-access-65m87\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.273950 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.273596 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.273950 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.273653 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.273950 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.273681 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.273950 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.273763 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.275541 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.275516 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dfe3317-eeae-4bf2-a116-a152249b4f59-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.275867 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.275841 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.275964 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.275916 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.276034 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.276015 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.285545 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.285524 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.285662 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.285558 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65m87\" (UniqueName: \"kubernetes.io/projected/9dfe3317-eeae-4bf2-a116-a152249b4f59-kube-api-access-65m87\") pod \"istiod-openshift-gateway-7876bfcb74-9m8js\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.379685 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.379594 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:19.658564 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.658487 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js"] Nov 03 20:03:19.662246 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:03:19.662220 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dfe3317_eeae_4bf2_a116_a152249b4f59.slice/crio-5f097ee2216d120fe77ac2052c6b069c8a5a9a9fe03508258a14a70705accf8f WatchSource:0}: Error finding container 5f097ee2216d120fe77ac2052c6b069c8a5a9a9fe03508258a14a70705accf8f: Status 404 returned error can't find the container with id 5f097ee2216d120fe77ac2052c6b069c8a5a9a9fe03508258a14a70705accf8f Nov 03 20:03:19.664098 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.664083 4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:03:19.742434 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:19.742398 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" event={"ID":"9dfe3317-eeae-4bf2-a116-a152249b4f59","Type":"ContainerStarted","Data":"5f097ee2216d120fe77ac2052c6b069c8a5a9a9fe03508258a14a70705accf8f"} Nov 03 20:03:22.962060 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:22.962022 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:03:22.962464 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:22.962104 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:03:23.760089 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:23.760050 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" event={"ID":"9dfe3317-eeae-4bf2-a116-a152249b4f59","Type":"ContainerStarted","Data":"487d5342cecb577475201bb05798b031097ebb051c803e548b997d4fdec870c3"} Nov 03 20:03:23.760296 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:23.760219 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:23.784754 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:23.784673 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" podStartSLOduration=1.487648225 podStartE2EDuration="4.784661304s" totalImagesPullingTime="3.297013079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:23.783368207 +0000 UTC m=+386.715235720" watchObservedRunningTime="2026-11-03 20:03:23.784661304 +0000 UTC m=+386.716528814" Nov 03 20:03:24.765952 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:03:24.765923 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:03:28.206567 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:03:28.206532 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:03:58.205860 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:03:58.205821 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:04:18.165312 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.165268 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc"] Nov 03 20:04:18.168856 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.168831 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:18.187409 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.187381 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 20:04:18.187667 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.187650 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 20:04:18.188548 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.188524 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 20:04:18.188681 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.188555 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 20:04:18.188681 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.188598 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 20:04:18.188681 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.188625 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-t7ddc\"" Nov 03 20:04:18.207294 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.207269 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc"] Nov 03 20:04:18.252226 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.252196 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1317f85-fde1-417b-a0aa-fe15302cfb9f-tmp\") pod \"opendatahub-operator-controller-manager-79746d584b-c2cmc\" (UID: \"c1317f85-fde1-417b-a0aa-fe15302cfb9f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:18.252364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.252236 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1317f85-fde1-417b-a0aa-fe15302cfb9f-metrics-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-c2cmc\" (UID: \"c1317f85-fde1-417b-a0aa-fe15302cfb9f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:18.252364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.252258 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpl5f\" (UniqueName: \"kubernetes.io/projected/c1317f85-fde1-417b-a0aa-fe15302cfb9f-kube-api-access-tpl5f\") pod \"opendatahub-operator-controller-manager-79746d584b-c2cmc\" (UID: \"c1317f85-fde1-417b-a0aa-fe15302cfb9f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:18.252364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.252330 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c1317f85-fde1-417b-a0aa-fe15302cfb9f-webhook-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-c2cmc\" (UID: \"c1317f85-fde1-417b-a0aa-fe15302cfb9f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:18.252521 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.252365 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c1317f85-fde1-417b-a0aa-fe15302cfb9f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-c2cmc\" (UID: \"c1317f85-fde1-417b-a0aa-fe15302cfb9f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:18.252521 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.252396 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c1317f85-fde1-417b-a0aa-fe15302cfb9f-manifests\") pod \"opendatahub-operator-controller-manager-79746d584b-c2cmc\" (UID: \"c1317f85-fde1-417b-a0aa-fe15302cfb9f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:18.352793 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.352756 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c1317f85-fde1-417b-a0aa-fe15302cfb9f-webhook-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-c2cmc\" (UID: \"c1317f85-fde1-417b-a0aa-fe15302cfb9f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:18.352968 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.352805 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c1317f85-fde1-417b-a0aa-fe15302cfb9f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-c2cmc\" (UID: \"c1317f85-fde1-417b-a0aa-fe15302cfb9f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:18.352968 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.352826 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c1317f85-fde1-417b-a0aa-fe15302cfb9f-manifests\") pod \"opendatahub-operator-controller-manager-79746d584b-c2cmc\" (UID: \"c1317f85-fde1-417b-a0aa-fe15302cfb9f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:18.352968 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.352866 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1317f85-fde1-417b-a0aa-fe15302cfb9f-tmp\") pod \"opendatahub-operator-controller-manager-79746d584b-c2cmc\" (UID: \"c1317f85-fde1-417b-a0aa-fe15302cfb9f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:18.352968 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.352889 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1317f85-fde1-417b-a0aa-fe15302cfb9f-metrics-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-c2cmc\" (UID: \"c1317f85-fde1-417b-a0aa-fe15302cfb9f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:18.352968 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.352911 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpl5f\" (UniqueName: \"kubernetes.io/projected/c1317f85-fde1-417b-a0aa-fe15302cfb9f-kube-api-access-tpl5f\") pod \"opendatahub-operator-controller-manager-79746d584b-c2cmc\" (UID: \"c1317f85-fde1-417b-a0aa-fe15302cfb9f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:18.353415 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.353385 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c1317f85-fde1-417b-a0aa-fe15302cfb9f-manifests\") pod \"opendatahub-operator-controller-manager-79746d584b-c2cmc\" (UID: \"c1317f85-fde1-417b-a0aa-fe15302cfb9f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:18.355861 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.355835 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1317f85-fde1-417b-a0aa-fe15302cfb9f-tmp\") pod \"opendatahub-operator-controller-manager-79746d584b-c2cmc\" (UID: \"c1317f85-fde1-417b-a0aa-fe15302cfb9f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:18.355993 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.355970 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c1317f85-fde1-417b-a0aa-fe15302cfb9f-webhook-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-c2cmc\" (UID: \"c1317f85-fde1-417b-a0aa-fe15302cfb9f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:18.356040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.356026 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1317f85-fde1-417b-a0aa-fe15302cfb9f-metrics-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-c2cmc\" (UID: \"c1317f85-fde1-417b-a0aa-fe15302cfb9f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:18.356136 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.356116 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c1317f85-fde1-417b-a0aa-fe15302cfb9f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-c2cmc\" (UID: \"c1317f85-fde1-417b-a0aa-fe15302cfb9f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:18.366611 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.366587 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpl5f\" (UniqueName: \"kubernetes.io/projected/c1317f85-fde1-417b-a0aa-fe15302cfb9f-kube-api-access-tpl5f\") pod \"opendatahub-operator-controller-manager-79746d584b-c2cmc\" (UID: \"c1317f85-fde1-417b-a0aa-fe15302cfb9f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:18.483404 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.483316 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:18.789462 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.789365 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc"] Nov 03 20:04:18.795777 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:04:18.795749 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1317f85_fde1_417b_a0aa_fe15302cfb9f.slice/crio-fca9d31c0fa680a76ca57c1b9eba20f891c74e21c8248f64cc2c66d11e8f4d7c WatchSource:0}: Error finding container fca9d31c0fa680a76ca57c1b9eba20f891c74e21c8248f64cc2c66d11e8f4d7c: Status 404 returned error can't find the container with id fca9d31c0fa680a76ca57c1b9eba20f891c74e21c8248f64cc2c66d11e8f4d7c Nov 03 20:04:18.974386 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:18.974348 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" event={"ID":"c1317f85-fde1-417b-a0aa-fe15302cfb9f","Type":"ContainerStarted","Data":"fca9d31c0fa680a76ca57c1b9eba20f891c74e21c8248f64cc2c66d11e8f4d7c"} Nov 03 20:04:22.993460 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:22.993422 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" event={"ID":"c1317f85-fde1-417b-a0aa-fe15302cfb9f","Type":"ContainerStarted","Data":"015bf19058434779ada480997a2a93ccc57da72e9a491f21806b048bbdf41475"} Nov 03 20:04:23.998935 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:23.998889 4918 generic.go:356] "Generic (PLEG): container finished" podID="c1317f85-fde1-417b-a0aa-fe15302cfb9f" containerID="015bf19058434779ada480997a2a93ccc57da72e9a491f21806b048bbdf41475" exitCode=0 Nov 03 20:04:23.999390 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:23.998975 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" event={"ID":"c1317f85-fde1-417b-a0aa-fe15302cfb9f","Type":"ContainerDied","Data":"015bf19058434779ada480997a2a93ccc57da72e9a491f21806b048bbdf41475"} Nov 03 20:04:25.004352 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:25.004315 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" event={"ID":"c1317f85-fde1-417b-a0aa-fe15302cfb9f","Type":"ContainerStarted","Data":"d1dd54b85c52d38d3f93a4b7577d792478b7c5deb0d65ee6ebc9a9e52a068b12"} Nov 03 20:04:25.004870 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:25.004452 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:25.095972 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:25.095913 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" podStartSLOduration=2.210700116 podStartE2EDuration="7.09589731s" totalImagesPullingTime="4.399496736s" totalInitContainerRuntime="485.700458ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:25.093862623 +0000 UTC m=+448.025730148" watchObservedRunningTime="2026-11-03 20:04:25.09589731 +0000 UTC m=+448.027764823" Nov 03 20:04:28.206632 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:04:28.206592 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:04:36.011171 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:04:36.011136 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-c2cmc" Nov 03 20:04:58.206581 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:04:58.206544 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:05:00.982280 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:00.982244 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-667774d4b7-l8npv"] Nov 03 20:05:00.986779 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:00.986755 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-667774d4b7-l8npv" Nov 03 20:05:00.999642 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:00.999617 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 20:05:00.999848 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:00.999689 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 20:05:00.999848 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:00.999761 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jkmgq\"" Nov 03 20:05:01.007817 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:01.007790 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-667774d4b7-l8npv"] Nov 03 20:05:01.071280 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:01.071244 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0000aa42-8c87-45d2-990d-c69f4d9c0ff9-tls-certs\") pod \"kube-auth-proxy-667774d4b7-l8npv\" (UID: \"0000aa42-8c87-45d2-990d-c69f4d9c0ff9\") " pod="openshift-ingress/kube-auth-proxy-667774d4b7-l8npv" Nov 03 20:05:01.071442 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:01.071291 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trxl9\" (UniqueName: \"kubernetes.io/projected/0000aa42-8c87-45d2-990d-c69f4d9c0ff9-kube-api-access-trxl9\") pod \"kube-auth-proxy-667774d4b7-l8npv\" (UID: \"0000aa42-8c87-45d2-990d-c69f4d9c0ff9\") " pod="openshift-ingress/kube-auth-proxy-667774d4b7-l8npv" Nov 03 20:05:01.071442 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:01.071406 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0000aa42-8c87-45d2-990d-c69f4d9c0ff9-tmp\") pod \"kube-auth-proxy-667774d4b7-l8npv\" (UID: \"0000aa42-8c87-45d2-990d-c69f4d9c0ff9\") " pod="openshift-ingress/kube-auth-proxy-667774d4b7-l8npv" Nov 03 20:05:01.172238 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:01.172204 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0000aa42-8c87-45d2-990d-c69f4d9c0ff9-tmp\") pod \"kube-auth-proxy-667774d4b7-l8npv\" (UID: \"0000aa42-8c87-45d2-990d-c69f4d9c0ff9\") " pod="openshift-ingress/kube-auth-proxy-667774d4b7-l8npv" Nov 03 20:05:01.172238 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:01.172244 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0000aa42-8c87-45d2-990d-c69f4d9c0ff9-tls-certs\") pod \"kube-auth-proxy-667774d4b7-l8npv\" (UID: \"0000aa42-8c87-45d2-990d-c69f4d9c0ff9\") " pod="openshift-ingress/kube-auth-proxy-667774d4b7-l8npv" Nov 03 20:05:01.172453 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:01.172281 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trxl9\" (UniqueName: \"kubernetes.io/projected/0000aa42-8c87-45d2-990d-c69f4d9c0ff9-kube-api-access-trxl9\") pod \"kube-auth-proxy-667774d4b7-l8npv\" (UID: \"0000aa42-8c87-45d2-990d-c69f4d9c0ff9\") " pod="openshift-ingress/kube-auth-proxy-667774d4b7-l8npv" Nov 03 20:05:01.174540 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:01.174513 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0000aa42-8c87-45d2-990d-c69f4d9c0ff9-tmp\") pod \"kube-auth-proxy-667774d4b7-l8npv\" (UID: \"0000aa42-8c87-45d2-990d-c69f4d9c0ff9\") " pod="openshift-ingress/kube-auth-proxy-667774d4b7-l8npv" Nov 03 20:05:01.174649 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:01.174608 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0000aa42-8c87-45d2-990d-c69f4d9c0ff9-tls-certs\") pod \"kube-auth-proxy-667774d4b7-l8npv\" (UID: \"0000aa42-8c87-45d2-990d-c69f4d9c0ff9\") " pod="openshift-ingress/kube-auth-proxy-667774d4b7-l8npv" Nov 03 20:05:01.189016 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:01.188984 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trxl9\" (UniqueName: \"kubernetes.io/projected/0000aa42-8c87-45d2-990d-c69f4d9c0ff9-kube-api-access-trxl9\") pod \"kube-auth-proxy-667774d4b7-l8npv\" (UID: \"0000aa42-8c87-45d2-990d-c69f4d9c0ff9\") " pod="openshift-ingress/kube-auth-proxy-667774d4b7-l8npv" Nov 03 20:05:01.297576 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:01.297496 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-667774d4b7-l8npv" Nov 03 20:05:01.573694 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:01.573668 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-667774d4b7-l8npv"] Nov 03 20:05:01.575917 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:05:01.575897 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0000aa42_8c87_45d2_990d_c69f4d9c0ff9.slice/crio-497879f1ad6c8f091685f0d02ada8bf6bb8765c9d8ce669e51d47860eadae352 WatchSource:0}: Error finding container 497879f1ad6c8f091685f0d02ada8bf6bb8765c9d8ce669e51d47860eadae352: Status 404 returned error can't find the container with id 497879f1ad6c8f091685f0d02ada8bf6bb8765c9d8ce669e51d47860eadae352 Nov 03 20:05:02.149376 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:02.149340 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-667774d4b7-l8npv" event={"ID":"0000aa42-8c87-45d2-990d-c69f4d9c0ff9","Type":"ContainerStarted","Data":"497879f1ad6c8f091685f0d02ada8bf6bb8765c9d8ce669e51d47860eadae352"} Nov 03 20:05:04.159089 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.159048 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-667774d4b7-l8npv" event={"ID":"0000aa42-8c87-45d2-990d-c69f4d9c0ff9","Type":"ContainerStarted","Data":"e8b274d199beed6848903dcae4a69df3b4930cb6d60a841ab026a671c0ecbc81"} Nov 03 20:05:04.178866 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.178753 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-667774d4b7-l8npv" podStartSLOduration=1.833619518 podStartE2EDuration="4.178739141s" totalImagesPullingTime="2.345119623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:04.177480449 +0000 UTC m=+487.109347960" watchObservedRunningTime="2026-11-03 20:05:04.178739141 +0000 UTC m=+487.110606682" Nov 03 20:05:04.329359 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.329325 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5"] Nov 03 20:05:04.332722 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.332701 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5" Nov 03 20:05:04.339410 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.339383 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 20:05:04.339542 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.339446 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 20:05:04.339602 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.339555 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-2dgnh\"" Nov 03 20:05:04.344120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.344080 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 20:05:04.345370 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.345346 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5"] Nov 03 20:05:04.401608 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.401572 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e07b0f-b78b-4f51-9ead-028f98aec0c9-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-dbzn5\" (UID: \"60e07b0f-b78b-4f51-9ead-028f98aec0c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5" Nov 03 20:05:04.401608 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.401610 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/60e07b0f-b78b-4f51-9ead-028f98aec0c9-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-dbzn5\" (UID: \"60e07b0f-b78b-4f51-9ead-028f98aec0c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5" Nov 03 20:05:04.401815 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.401711 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60e07b0f-b78b-4f51-9ead-028f98aec0c9-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-dbzn5\" (UID: \"60e07b0f-b78b-4f51-9ead-028f98aec0c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5" Nov 03 20:05:04.401815 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.401755 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4kjr\" (UniqueName: \"kubernetes.io/projected/60e07b0f-b78b-4f51-9ead-028f98aec0c9-kube-api-access-t4kjr\") pod \"kserve-module-controller-manager-7566cf5fc8-dbzn5\" (UID: \"60e07b0f-b78b-4f51-9ead-028f98aec0c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5" Nov 03 20:05:04.502728 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.502641 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60e07b0f-b78b-4f51-9ead-028f98aec0c9-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-dbzn5\" (UID: \"60e07b0f-b78b-4f51-9ead-028f98aec0c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5" Nov 03 20:05:04.502728 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.502686 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4kjr\" (UniqueName: \"kubernetes.io/projected/60e07b0f-b78b-4f51-9ead-028f98aec0c9-kube-api-access-t4kjr\") pod \"kserve-module-controller-manager-7566cf5fc8-dbzn5\" (UID: \"60e07b0f-b78b-4f51-9ead-028f98aec0c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5" Nov 03 20:05:04.502931 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.502807 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e07b0f-b78b-4f51-9ead-028f98aec0c9-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-dbzn5\" (UID: \"60e07b0f-b78b-4f51-9ead-028f98aec0c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5" Nov 03 20:05:04.502931 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.502862 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/60e07b0f-b78b-4f51-9ead-028f98aec0c9-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-dbzn5\" (UID: \"60e07b0f-b78b-4f51-9ead-028f98aec0c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5" Nov 03 20:05:04.503249 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.503229 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e07b0f-b78b-4f51-9ead-028f98aec0c9-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-dbzn5\" (UID: \"60e07b0f-b78b-4f51-9ead-028f98aec0c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5" Nov 03 20:05:04.503290 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.503264 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/60e07b0f-b78b-4f51-9ead-028f98aec0c9-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-dbzn5\" (UID: \"60e07b0f-b78b-4f51-9ead-028f98aec0c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5" Nov 03 20:05:04.503588 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.503568 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60e07b0f-b78b-4f51-9ead-028f98aec0c9-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-dbzn5\" (UID: \"60e07b0f-b78b-4f51-9ead-028f98aec0c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5" Nov 03 20:05:04.515351 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.515321 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4kjr\" (UniqueName: \"kubernetes.io/projected/60e07b0f-b78b-4f51-9ead-028f98aec0c9-kube-api-access-t4kjr\") pod \"kserve-module-controller-manager-7566cf5fc8-dbzn5\" (UID: \"60e07b0f-b78b-4f51-9ead-028f98aec0c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5" Nov 03 20:05:04.645362 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.645324 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5" Nov 03 20:05:04.922939 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:04.922577 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5"] Nov 03 20:05:04.925246 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:05:04.925221 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e07b0f_b78b_4f51_9ead_028f98aec0c9.slice/crio-02f0946986111fdf1df7b8ae10aa7083123209871bcc1927c853f909bf2fa8d1 WatchSource:0}: Error finding container 02f0946986111fdf1df7b8ae10aa7083123209871bcc1927c853f909bf2fa8d1: Status 404 returned error can't find the container with id 02f0946986111fdf1df7b8ae10aa7083123209871bcc1927c853f909bf2fa8d1 Nov 03 20:05:05.164470 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:05.164431 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5" event={"ID":"60e07b0f-b78b-4f51-9ead-028f98aec0c9","Type":"ContainerStarted","Data":"02f0946986111fdf1df7b8ae10aa7083123209871bcc1927c853f909bf2fa8d1"} Nov 03 20:05:07.174511 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:07.174475 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5" event={"ID":"60e07b0f-b78b-4f51-9ead-028f98aec0c9","Type":"ContainerStarted","Data":"74ea72bf536298444d1772b407dc9ee87776db722b12e8d88ae28714644c414e"} Nov 03 20:05:07.174924 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:07.174553 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5" Nov 03 20:05:07.202967 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:07.199432 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5" podStartSLOduration=1.7080925850000002 podStartE2EDuration="3.199408163s" totalImagesPullingTime="1.491315578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:07.197227754 +0000 UTC m=+490.129095267" watchObservedRunningTime="2026-11-03 20:05:07.199408163 +0000 UTC m=+490.131275672" Nov 03 20:05:28.179313 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:28.179280 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dbzn5" Nov 03 20:05:28.207310 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:05:28.207273 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:05:33.661505 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.661465 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj"] Nov 03 20:05:33.664951 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.664930 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj" Nov 03 20:05:33.669802 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.669783 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 20:05:33.671853 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.671835 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-98hrh\"" Nov 03 20:05:33.671952 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.671885 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 20:05:33.671952 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.671885 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 20:05:33.674669 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.674593 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:05:33.676013 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.675988 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj"] Nov 03 20:05:33.737994 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.737953 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e796581-1271-464f-a3b9-8c37c0f544af-cert\") pod \"jobset-controller-manager-55fc7c68c4-8zwzj\" (UID: \"7e796581-1271-464f-a3b9-8c37c0f544af\") " pod="openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj" Nov 03 20:05:33.738169 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.738072 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7e796581-1271-464f-a3b9-8c37c0f544af-manager-config\") pod \"jobset-controller-manager-55fc7c68c4-8zwzj\" (UID: \"7e796581-1271-464f-a3b9-8c37c0f544af\") " pod="openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj" Nov 03 20:05:33.738169 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.738096 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flrcb\" (UniqueName: \"kubernetes.io/projected/7e796581-1271-464f-a3b9-8c37c0f544af-kube-api-access-flrcb\") pod \"jobset-controller-manager-55fc7c68c4-8zwzj\" (UID: \"7e796581-1271-464f-a3b9-8c37c0f544af\") " pod="openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj" Nov 03 20:05:33.738283 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.738227 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e796581-1271-464f-a3b9-8c37c0f544af-metrics-certs\") pod \"jobset-controller-manager-55fc7c68c4-8zwzj\" (UID: \"7e796581-1271-464f-a3b9-8c37c0f544af\") " pod="openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj" Nov 03 20:05:33.772144 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.772113 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 20:05:33.839055 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.839021 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7e796581-1271-464f-a3b9-8c37c0f544af-manager-config\") pod \"jobset-controller-manager-55fc7c68c4-8zwzj\" (UID: \"7e796581-1271-464f-a3b9-8c37c0f544af\") " pod="openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj" Nov 03 20:05:33.839229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.839060 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flrcb\" (UniqueName: \"kubernetes.io/projected/7e796581-1271-464f-a3b9-8c37c0f544af-kube-api-access-flrcb\") pod \"jobset-controller-manager-55fc7c68c4-8zwzj\" (UID: \"7e796581-1271-464f-a3b9-8c37c0f544af\") " pod="openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj" Nov 03 20:05:33.839229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.839108 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e796581-1271-464f-a3b9-8c37c0f544af-metrics-certs\") pod \"jobset-controller-manager-55fc7c68c4-8zwzj\" (UID: \"7e796581-1271-464f-a3b9-8c37c0f544af\") " pod="openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj" Nov 03 20:05:33.839229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.839128 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e796581-1271-464f-a3b9-8c37c0f544af-cert\") pod \"jobset-controller-manager-55fc7c68c4-8zwzj\" (UID: \"7e796581-1271-464f-a3b9-8c37c0f544af\") " pod="openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj" Nov 03 20:05:33.839814 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.839780 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7e796581-1271-464f-a3b9-8c37c0f544af-manager-config\") pod \"jobset-controller-manager-55fc7c68c4-8zwzj\" (UID: \"7e796581-1271-464f-a3b9-8c37c0f544af\") " pod="openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj" Nov 03 20:05:33.841696 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.841675 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e796581-1271-464f-a3b9-8c37c0f544af-metrics-certs\") pod \"jobset-controller-manager-55fc7c68c4-8zwzj\" (UID: \"7e796581-1271-464f-a3b9-8c37c0f544af\") " pod="openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj" Nov 03 20:05:33.841783 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.841750 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e796581-1271-464f-a3b9-8c37c0f544af-cert\") pod \"jobset-controller-manager-55fc7c68c4-8zwzj\" (UID: \"7e796581-1271-464f-a3b9-8c37c0f544af\") " pod="openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj" Nov 03 20:05:33.852207 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.852164 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flrcb\" (UniqueName: \"kubernetes.io/projected/7e796581-1271-464f-a3b9-8c37c0f544af-kube-api-access-flrcb\") pod \"jobset-controller-manager-55fc7c68c4-8zwzj\" (UID: \"7e796581-1271-464f-a3b9-8c37c0f544af\") " pod="openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj" Nov 03 20:05:33.977326 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:33.977244 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj" Nov 03 20:05:34.253262 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:34.253146 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj"] Nov 03 20:05:34.255892 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:05:34.255866 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e796581_1271_464f_a3b9_8c37c0f544af.slice/crio-6cea53842df7ebf7f5bcaf0a24380a04d6022f1003e204fd86ef8ff67e8d72ff WatchSource:0}: Error finding container 6cea53842df7ebf7f5bcaf0a24380a04d6022f1003e204fd86ef8ff67e8d72ff: Status 404 returned error can't find the container with id 6cea53842df7ebf7f5bcaf0a24380a04d6022f1003e204fd86ef8ff67e8d72ff Nov 03 20:05:34.285084 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:34.285053 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj" event={"ID":"7e796581-1271-464f-a3b9-8c37c0f544af","Type":"ContainerStarted","Data":"6cea53842df7ebf7f5bcaf0a24380a04d6022f1003e204fd86ef8ff67e8d72ff"} Nov 03 20:05:38.303193 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:38.303149 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj" event={"ID":"7e796581-1271-464f-a3b9-8c37c0f544af","Type":"ContainerStarted","Data":"f52a5dfe44c9cfe414406ed9da07210fb58636a09984c7b94a6f41188fbf9666"} Nov 03 20:05:38.303606 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:38.303243 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj" Nov 03 20:05:49.313493 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:49.313460 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj" Nov 03 20:05:49.334915 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:05:49.334853 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-55fc7c68c4-8zwzj" podStartSLOduration=12.402128628 podStartE2EDuration="16.334834592s" totalImagesPullingTime="3.932705964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:38.323578499 +0000 UTC m=+521.255446011" watchObservedRunningTime="2026-11-03 20:05:49.334834592 +0000 UTC m=+532.266702105" Nov 03 20:05:58.207509 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:05:58.207471 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:06:03.607534 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:03.607497 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js"] Nov 03 20:06:03.608038 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:03.607696 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" podUID="9dfe3317-eeae-4bf2-a116-a152249b4f59" containerName="discovery" containerID="cri-o://487d5342cecb577475201bb05798b031097ebb051c803e548b997d4fdec870c3" gracePeriod=30 Nov 03 20:06:03.765356 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:03.765319 4918 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-9m8js container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.27:8080/ready\": dial tcp 10.132.0.27:8080: connect: connection refused" start-of-body= Nov 03 20:06:03.765508 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:03.765386 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" podUID="9dfe3317-eeae-4bf2-a116-a152249b4f59" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.27:8080/ready\": dial tcp 10.132.0.27:8080: connect: connection refused" Nov 03 20:06:03.847990 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:03.847963 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:06:03.977331 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:03.977248 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-token\" (UniqueName: \"kubernetes.io/projected/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-token\") pod \"9dfe3317-eeae-4bf2-a116-a152249b4f59\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " Nov 03 20:06:03.977331 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:03.977293 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dfe3317-eeae-4bf2-a116-a152249b4f59-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dfe3317-eeae-4bf2-a116-a152249b4f59-local-certs\") pod \"9dfe3317-eeae-4bf2-a116-a152249b4f59\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " Nov 03 20:06:03.977563 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:03.977336 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-kubeconfig\") pod \"9dfe3317-eeae-4bf2-a116-a152249b4f59\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " Nov 03 20:06:03.977563 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:03.977372 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-csr-ca-configmap\") pod \"9dfe3317-eeae-4bf2-a116-a152249b4f59\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " Nov 03 20:06:03.977563 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:03.977415 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dfe3317-eeae-4bf2-a116-a152249b4f59-kube-api-access-65m87\" (UniqueName: \"kubernetes.io/projected/9dfe3317-eeae-4bf2-a116-a152249b4f59-kube-api-access-65m87\") pod \"9dfe3317-eeae-4bf2-a116-a152249b4f59\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " Nov 03 20:06:03.977563 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:03.977441 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-csr-dns-cert\") pod \"9dfe3317-eeae-4bf2-a116-a152249b4f59\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " Nov 03 20:06:03.977563 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:03.977499 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-cacerts\" (UniqueName: \"kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-cacerts\") pod \"9dfe3317-eeae-4bf2-a116-a152249b4f59\" (UID: \"9dfe3317-eeae-4bf2-a116-a152249b4f59\") " Nov 03 20:06:03.977817 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:03.977786 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-csr-ca-configmap" pod "9dfe3317-eeae-4bf2-a116-a152249b4f59" (UID: "9dfe3317-eeae-4bf2-a116-a152249b4f59"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:03.979688 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:03.979645 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfe3317-eeae-4bf2-a116-a152249b4f59-local-certs" pod "9dfe3317-eeae-4bf2-a116-a152249b4f59" (UID: "9dfe3317-eeae-4bf2-a116-a152249b4f59"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:03.979799 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:03.979744 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-kubeconfig" pod "9dfe3317-eeae-4bf2-a116-a152249b4f59" (UID: "9dfe3317-eeae-4bf2-a116-a152249b4f59"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:03.979918 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:03.979900 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-token" pod "9dfe3317-eeae-4bf2-a116-a152249b4f59" (UID: "9dfe3317-eeae-4bf2-a116-a152249b4f59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:03.979984 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:03.979932 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dfe3317-eeae-4bf2-a116-a152249b4f59-kube-api-access-65m87" pod "9dfe3317-eeae-4bf2-a116-a152249b4f59" (UID: "9dfe3317-eeae-4bf2-a116-a152249b4f59"). InnerVolumeSpecName "kube-api-access-65m87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:03.980079 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:03.980063 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-cacerts" pod "9dfe3317-eeae-4bf2-a116-a152249b4f59" (UID: "9dfe3317-eeae-4bf2-a116-a152249b4f59"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:03.980138 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:03.980111 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-csr-dns-cert" pod "9dfe3317-eeae-4bf2-a116-a152249b4f59" (UID: "9dfe3317-eeae-4bf2-a116-a152249b4f59"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:04.078838 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:04.078777 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65m87\" (UniqueName: \"kubernetes.io/projected/9dfe3317-eeae-4bf2-a116-a152249b4f59-kube-api-access-65m87\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:04.078838 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:04.078817 4918 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-csr-dns-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:04.078838 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:04.078832 4918 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-cacerts\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:04.079067 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:04.078844 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:04.079067 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:04.078859 4918 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dfe3317-eeae-4bf2-a116-a152249b4f59-local-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:04.079067 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:04.078872 4918 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-kubeconfig\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:04.079067 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:04.078884 4918 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9dfe3317-eeae-4bf2-a116-a152249b4f59-istio-csr-ca-configmap\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:04.420398 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:04.420362 4918 generic.go:356] "Generic (PLEG): container finished" podID="9dfe3317-eeae-4bf2-a116-a152249b4f59" containerID="487d5342cecb577475201bb05798b031097ebb051c803e548b997d4fdec870c3" exitCode=0 Nov 03 20:06:04.420588 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:04.420407 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" Nov 03 20:06:04.420588 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:04.420448 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" event={"ID":"9dfe3317-eeae-4bf2-a116-a152249b4f59","Type":"ContainerDied","Data":"487d5342cecb577475201bb05798b031097ebb051c803e548b997d4fdec870c3"} Nov 03 20:06:04.420588 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:04.420497 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js" event={"ID":"9dfe3317-eeae-4bf2-a116-a152249b4f59","Type":"ContainerDied","Data":"5f097ee2216d120fe77ac2052c6b069c8a5a9a9fe03508258a14a70705accf8f"} Nov 03 20:06:04.420588 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:04.420516 4918 scope.go:122] "RemoveContainer" containerID="487d5342cecb577475201bb05798b031097ebb051c803e548b997d4fdec870c3" Nov 03 20:06:04.429240 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:04.429222 4918 scope.go:122] "RemoveContainer" containerID="487d5342cecb577475201bb05798b031097ebb051c803e548b997d4fdec870c3" Nov 03 20:06:04.429480 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:06:04.429460 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"487d5342cecb577475201bb05798b031097ebb051c803e548b997d4fdec870c3\": container with ID starting with 487d5342cecb577475201bb05798b031097ebb051c803e548b997d4fdec870c3 not found: ID does not exist" containerID="487d5342cecb577475201bb05798b031097ebb051c803e548b997d4fdec870c3" Nov 03 20:06:04.429542 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:04.429487 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"487d5342cecb577475201bb05798b031097ebb051c803e548b997d4fdec870c3"} err="failed to get container status \"487d5342cecb577475201bb05798b031097ebb051c803e548b997d4fdec870c3\": rpc error: code = NotFound desc = could not find container \"487d5342cecb577475201bb05798b031097ebb051c803e548b997d4fdec870c3\": container with ID starting with 487d5342cecb577475201bb05798b031097ebb051c803e548b997d4fdec870c3 not found: ID does not exist" Nov 03 20:06:04.442722 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:04.442685 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js"] Nov 03 20:06:04.447615 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:04.447588 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-9m8js"] Nov 03 20:06:05.213108 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:05.213075 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dfe3317-eeae-4bf2-a116-a152249b4f59" path="/var/lib/kubelet/pods/9dfe3317-eeae-4bf2-a116-a152249b4f59/volumes" Nov 03 20:06:21.594082 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.594039 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg"] Nov 03 20:06:21.594588 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.594572 4918 cpu_manager.go:414] "containerMap: removing container" podUID="9dfe3317-eeae-4bf2-a116-a152249b4f59" containerName="discovery" Nov 03 20:06:21.594664 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.594592 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dfe3317-eeae-4bf2-a116-a152249b4f59" containerName="discovery" Nov 03 20:06:21.594715 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.594677 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dfe3317-eeae-4bf2-a116-a152249b4f59" containerName="discovery" Nov 03 20:06:21.597330 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.597309 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" Nov 03 20:06:21.608705 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.608681 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:06:21.611720 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.611701 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:06:21.614155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.614124 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg"] Nov 03 20:06:21.616400 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.616378 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 20:06:21.616498 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.616470 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:06:21.616498 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.616477 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-znhr8\"" Nov 03 20:06:21.730814 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.730764 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ed8754d-0475-4c68-8c8c-c2db790f6c89-maas-api-tls\") pod \"maas-api-6d669c7fcd-sxphg\" (UID: \"0ed8754d-0475-4c68-8c8c-c2db790f6c89\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" Nov 03 20:06:21.730983 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.730829 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtsxq\" (UniqueName: \"kubernetes.io/projected/0ed8754d-0475-4c68-8c8c-c2db790f6c89-kube-api-access-qtsxq\") pod \"maas-api-6d669c7fcd-sxphg\" (UID: \"0ed8754d-0475-4c68-8c8c-c2db790f6c89\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" Nov 03 20:06:21.730983 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.730945 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed8754d-0475-4c68-8c8c-c2db790f6c89-openshift-service-ca\") pod \"maas-api-6d669c7fcd-sxphg\" (UID: \"0ed8754d-0475-4c68-8c8c-c2db790f6c89\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" Nov 03 20:06:21.731059 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.730994 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ed8754d-0475-4c68-8c8c-c2db790f6c89-metrics-certs\") pod \"maas-api-6d669c7fcd-sxphg\" (UID: \"0ed8754d-0475-4c68-8c8c-c2db790f6c89\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" Nov 03 20:06:21.832106 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.832067 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ed8754d-0475-4c68-8c8c-c2db790f6c89-maas-api-tls\") pod \"maas-api-6d669c7fcd-sxphg\" (UID: \"0ed8754d-0475-4c68-8c8c-c2db790f6c89\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" Nov 03 20:06:21.832296 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.832120 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtsxq\" (UniqueName: \"kubernetes.io/projected/0ed8754d-0475-4c68-8c8c-c2db790f6c89-kube-api-access-qtsxq\") pod \"maas-api-6d669c7fcd-sxphg\" (UID: \"0ed8754d-0475-4c68-8c8c-c2db790f6c89\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" Nov 03 20:06:21.832296 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.832145 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed8754d-0475-4c68-8c8c-c2db790f6c89-openshift-service-ca\") pod \"maas-api-6d669c7fcd-sxphg\" (UID: \"0ed8754d-0475-4c68-8c8c-c2db790f6c89\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" Nov 03 20:06:21.832296 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.832194 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ed8754d-0475-4c68-8c8c-c2db790f6c89-metrics-certs\") pod \"maas-api-6d669c7fcd-sxphg\" (UID: \"0ed8754d-0475-4c68-8c8c-c2db790f6c89\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" Nov 03 20:06:21.833094 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.833063 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed8754d-0475-4c68-8c8c-c2db790f6c89-openshift-service-ca\") pod \"maas-api-6d669c7fcd-sxphg\" (UID: \"0ed8754d-0475-4c68-8c8c-c2db790f6c89\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" Nov 03 20:06:21.834948 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.834928 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ed8754d-0475-4c68-8c8c-c2db790f6c89-metrics-certs\") pod \"maas-api-6d669c7fcd-sxphg\" (UID: \"0ed8754d-0475-4c68-8c8c-c2db790f6c89\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" Nov 03 20:06:21.835031 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.835011 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ed8754d-0475-4c68-8c8c-c2db790f6c89-maas-api-tls\") pod \"maas-api-6d669c7fcd-sxphg\" (UID: \"0ed8754d-0475-4c68-8c8c-c2db790f6c89\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" Nov 03 20:06:21.844319 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.844262 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtsxq\" (UniqueName: \"kubernetes.io/projected/0ed8754d-0475-4c68-8c8c-c2db790f6c89-kube-api-access-qtsxq\") pod \"maas-api-6d669c7fcd-sxphg\" (UID: \"0ed8754d-0475-4c68-8c8c-c2db790f6c89\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" Nov 03 20:06:21.909451 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:21.909406 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" Nov 03 20:06:22.185819 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:22.185670 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg"] Nov 03 20:06:22.188401 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:06:22.188374 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed8754d_0475_4c68_8c8c_c2db790f6c89.slice/crio-7634cd5fa06473d521df8b7179a78948b558d7038ea85fa15d67bc8c4a101815 WatchSource:0}: Error finding container 7634cd5fa06473d521df8b7179a78948b558d7038ea85fa15d67bc8c4a101815: Status 404 returned error can't find the container with id 7634cd5fa06473d521df8b7179a78948b558d7038ea85fa15d67bc8c4a101815 Nov 03 20:06:22.494668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:22.494579 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" event={"ID":"0ed8754d-0475-4c68-8c8c-c2db790f6c89","Type":"ContainerStarted","Data":"7634cd5fa06473d521df8b7179a78948b558d7038ea85fa15d67bc8c4a101815"} Nov 03 20:06:25.510092 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:25.510053 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" event={"ID":"0ed8754d-0475-4c68-8c8c-c2db790f6c89","Type":"ContainerStarted","Data":"ff1ea6747bf5683945149a9d2bef3110b1b181383288456ea92becb40508775b"} Nov 03 20:06:25.510564 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:25.510198 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" Nov 03 20:06:25.531301 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:25.531255 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" podStartSLOduration=2.218588269 podStartE2EDuration="4.53124206s" totalImagesPullingTime="2.312653791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:25.530230553 +0000 UTC m=+568.462098069" watchObservedRunningTime="2026-11-03 20:06:25.53124206 +0000 UTC m=+568.463109584" Nov 03 20:06:28.207199 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:06:28.207077 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:06:31.520873 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:31.520844 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" Nov 03 20:06:32.390733 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.390699 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-r24km"] Nov 03 20:06:32.393963 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.393946 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-r24km" Nov 03 20:06:32.403996 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.403971 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 20:06:32.404132 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.404094 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 20:06:32.404132 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.404109 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 20:06:32.404271 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.404166 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-grh7n\"" Nov 03 20:06:32.404339 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.404317 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-r24km"] Nov 03 20:06:32.475802 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.475773 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p"] Nov 03 20:06:32.478993 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.478977 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" Nov 03 20:06:32.490476 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.490451 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p"] Nov 03 20:06:32.529361 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.529328 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f55rw\" (UniqueName: \"kubernetes.io/projected/c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3-kube-api-access-f55rw\") pod \"authorino-6889cbd8c4-r24km\" (UID: \"c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3\") " pod="kuadrant-system/authorino-6889cbd8c4-r24km" Nov 03 20:06:32.529839 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.529366 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3-tls-cert\") pod \"authorino-6889cbd8c4-r24km\" (UID: \"c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3\") " pod="kuadrant-system/authorino-6889cbd8c4-r24km" Nov 03 20:06:32.630727 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.630692 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3-tls-cert\") pod \"authorino-6889cbd8c4-r24km\" (UID: \"c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3\") " pod="kuadrant-system/authorino-6889cbd8c4-r24km" Nov 03 20:06:32.630924 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.630733 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bb27585-623d-46bb-ae29-046c214e2252-maas-api-tls\") pod \"maas-api-6c684ff7f7-qrf9p\" (UID: \"3bb27585-623d-46bb-ae29-046c214e2252\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" Nov 03 20:06:32.630924 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.630763 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bb27585-623d-46bb-ae29-046c214e2252-metrics-certs\") pod \"maas-api-6c684ff7f7-qrf9p\" (UID: \"3bb27585-623d-46bb-ae29-046c214e2252\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" Nov 03 20:06:32.630924 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.630884 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bb27585-623d-46bb-ae29-046c214e2252-openshift-service-ca\") pod \"maas-api-6c684ff7f7-qrf9p\" (UID: \"3bb27585-623d-46bb-ae29-046c214e2252\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" Nov 03 20:06:32.631089 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.630926 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shprg\" (UniqueName: \"kubernetes.io/projected/3bb27585-623d-46bb-ae29-046c214e2252-kube-api-access-shprg\") pod \"maas-api-6c684ff7f7-qrf9p\" (UID: \"3bb27585-623d-46bb-ae29-046c214e2252\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" Nov 03 20:06:32.631089 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.630996 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f55rw\" (UniqueName: \"kubernetes.io/projected/c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3-kube-api-access-f55rw\") pod \"authorino-6889cbd8c4-r24km\" (UID: \"c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3\") " pod="kuadrant-system/authorino-6889cbd8c4-r24km" Nov 03 20:06:32.633204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.633169 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3-tls-cert\") pod \"authorino-6889cbd8c4-r24km\" (UID: \"c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3\") " pod="kuadrant-system/authorino-6889cbd8c4-r24km" Nov 03 20:06:32.642968 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.642913 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f55rw\" (UniqueName: \"kubernetes.io/projected/c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3-kube-api-access-f55rw\") pod \"authorino-6889cbd8c4-r24km\" (UID: \"c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3\") " pod="kuadrant-system/authorino-6889cbd8c4-r24km" Nov 03 20:06:32.664891 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.664867 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-r24km"] Nov 03 20:06:32.665099 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.665084 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-r24km" Nov 03 20:06:32.732358 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.732323 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bb27585-623d-46bb-ae29-046c214e2252-metrics-certs\") pod \"maas-api-6c684ff7f7-qrf9p\" (UID: \"3bb27585-623d-46bb-ae29-046c214e2252\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" Nov 03 20:06:32.732530 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.732374 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bb27585-623d-46bb-ae29-046c214e2252-openshift-service-ca\") pod \"maas-api-6c684ff7f7-qrf9p\" (UID: \"3bb27585-623d-46bb-ae29-046c214e2252\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" Nov 03 20:06:32.732530 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.732390 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shprg\" (UniqueName: \"kubernetes.io/projected/3bb27585-623d-46bb-ae29-046c214e2252-kube-api-access-shprg\") pod \"maas-api-6c684ff7f7-qrf9p\" (UID: \"3bb27585-623d-46bb-ae29-046c214e2252\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" Nov 03 20:06:32.732530 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.732436 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bb27585-623d-46bb-ae29-046c214e2252-maas-api-tls\") pod \"maas-api-6c684ff7f7-qrf9p\" (UID: \"3bb27585-623d-46bb-ae29-046c214e2252\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" Nov 03 20:06:32.733577 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.733553 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bb27585-623d-46bb-ae29-046c214e2252-openshift-service-ca\") pod \"maas-api-6c684ff7f7-qrf9p\" (UID: \"3bb27585-623d-46bb-ae29-046c214e2252\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" Nov 03 20:06:32.735064 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.734865 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bb27585-623d-46bb-ae29-046c214e2252-maas-api-tls\") pod \"maas-api-6c684ff7f7-qrf9p\" (UID: \"3bb27585-623d-46bb-ae29-046c214e2252\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" Nov 03 20:06:32.736508 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.736475 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bb27585-623d-46bb-ae29-046c214e2252-metrics-certs\") pod \"maas-api-6c684ff7f7-qrf9p\" (UID: \"3bb27585-623d-46bb-ae29-046c214e2252\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" Nov 03 20:06:32.746748 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.746708 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shprg\" (UniqueName: \"kubernetes.io/projected/3bb27585-623d-46bb-ae29-046c214e2252-kube-api-access-shprg\") pod \"maas-api-6c684ff7f7-qrf9p\" (UID: \"3bb27585-623d-46bb-ae29-046c214e2252\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" Nov 03 20:06:32.791595 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.791564 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" Nov 03 20:06:32.938768 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:32.938678 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-r24km"] Nov 03 20:06:32.942286 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:06:32.942255 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0ff5b31_5c89_4c2b_bbf7_5b81e68981b3.slice/crio-7157882f23d4156d9d572bb14ffed8d5848e4ea385f1b9d8f55e0ad0be967840 WatchSource:0}: Error finding container 7157882f23d4156d9d572bb14ffed8d5848e4ea385f1b9d8f55e0ad0be967840: Status 404 returned error can't find the container with id 7157882f23d4156d9d572bb14ffed8d5848e4ea385f1b9d8f55e0ad0be967840 Nov 03 20:06:33.067783 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:33.067756 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p"] Nov 03 20:06:33.070064 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:06:33.070033 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb27585_623d_46bb_ae29_046c214e2252.slice/crio-62955e577e1ae41a45963fbafcbcd8a002431ef20ff5dd24bea720aec173829e WatchSource:0}: Error finding container 62955e577e1ae41a45963fbafcbcd8a002431ef20ff5dd24bea720aec173829e: Status 404 returned error can't find the container with id 62955e577e1ae41a45963fbafcbcd8a002431ef20ff5dd24bea720aec173829e Nov 03 20:06:33.545753 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:33.545715 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" event={"ID":"3bb27585-623d-46bb-ae29-046c214e2252","Type":"ContainerStarted","Data":"23911820db778d4a8d1f046223ca32315a286935adf68d857414417346d6c301"} Nov 03 20:06:33.546261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:33.545777 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" event={"ID":"3bb27585-623d-46bb-ae29-046c214e2252","Type":"ContainerStarted","Data":"62955e577e1ae41a45963fbafcbcd8a002431ef20ff5dd24bea720aec173829e"} Nov 03 20:06:33.546261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:33.545810 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" Nov 03 20:06:33.547077 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:33.547051 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-r24km" event={"ID":"c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3","Type":"ContainerStarted","Data":"7157882f23d4156d9d572bb14ffed8d5848e4ea385f1b9d8f55e0ad0be967840"} Nov 03 20:06:33.565927 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:33.565863 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" podStartSLOduration=1.189683148 podStartE2EDuration="1.565844941s" totalImagesPullingTime="376.161793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:33.565395544 +0000 UTC m=+576.497263053" watchObservedRunningTime="2026-11-03 20:06:33.565844941 +0000 UTC m=+576.497712454" Nov 03 20:06:36.563766 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:36.563725 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-r24km" event={"ID":"c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3","Type":"ContainerStarted","Data":"ac2b82e2566b72cbef827bd81cdf7d7b00922917ab4eaf47fae07db85ab0fd8c"} Nov 03 20:06:36.564275 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:36.563804 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-r24km" podUID="c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3" containerName="authorino" containerID="cri-o://ac2b82e2566b72cbef827bd81cdf7d7b00922917ab4eaf47fae07db85ab0fd8c" gracePeriod=30 Nov 03 20:06:36.582953 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:36.582905 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-r24km" podStartSLOduration=1.636905235 podStartE2EDuration="4.582891926s" totalImagesPullingTime="2.945986691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:36.58113804 +0000 UTC m=+579.513005554" watchObservedRunningTime="2026-11-03 20:06:36.582891926 +0000 UTC m=+579.514759459" Nov 03 20:06:36.798149 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:36.798124 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-r24km" Nov 03 20:06:36.867510 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:36.867436 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3-tls-cert\" (UniqueName: \"kubernetes.io/secret/c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3-tls-cert\") pod \"c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3\" (UID: \"c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3\") " Nov 03 20:06:36.867510 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:36.867493 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3-kube-api-access-f55rw\" (UniqueName: \"kubernetes.io/projected/c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3-kube-api-access-f55rw\") pod \"c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3\" (UID: \"c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3\") " Nov 03 20:06:36.869742 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:36.869715 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3-kube-api-access-f55rw" pod "c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3" (UID: "c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3"). InnerVolumeSpecName "kube-api-access-f55rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:36.877330 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:36.877302 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3-tls-cert" pod "c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3" (UID: "c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:36.968779 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:36.968743 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f55rw\" (UniqueName: \"kubernetes.io/projected/c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3-kube-api-access-f55rw\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:36.968779 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:36.968773 4918 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3-tls-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:37.569571 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:37.569538 4918 generic.go:356] "Generic (PLEG): container finished" podID="c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3" containerID="ac2b82e2566b72cbef827bd81cdf7d7b00922917ab4eaf47fae07db85ab0fd8c" exitCode=0 Nov 03 20:06:37.570042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:37.569587 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-r24km" Nov 03 20:06:37.570042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:37.569613 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-r24km" event={"ID":"c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3","Type":"ContainerDied","Data":"ac2b82e2566b72cbef827bd81cdf7d7b00922917ab4eaf47fae07db85ab0fd8c"} Nov 03 20:06:37.570042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:37.569644 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-r24km" event={"ID":"c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3","Type":"ContainerDied","Data":"7157882f23d4156d9d572bb14ffed8d5848e4ea385f1b9d8f55e0ad0be967840"} Nov 03 20:06:37.570042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:37.569660 4918 scope.go:122] "RemoveContainer" containerID="ac2b82e2566b72cbef827bd81cdf7d7b00922917ab4eaf47fae07db85ab0fd8c" Nov 03 20:06:37.580515 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:37.580495 4918 scope.go:122] "RemoveContainer" containerID="ac2b82e2566b72cbef827bd81cdf7d7b00922917ab4eaf47fae07db85ab0fd8c" Nov 03 20:06:37.580793 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:06:37.580774 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac2b82e2566b72cbef827bd81cdf7d7b00922917ab4eaf47fae07db85ab0fd8c\": container with ID starting with ac2b82e2566b72cbef827bd81cdf7d7b00922917ab4eaf47fae07db85ab0fd8c not found: ID does not exist" containerID="ac2b82e2566b72cbef827bd81cdf7d7b00922917ab4eaf47fae07db85ab0fd8c" Nov 03 20:06:37.580836 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:37.580805 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac2b82e2566b72cbef827bd81cdf7d7b00922917ab4eaf47fae07db85ab0fd8c"} err="failed to get container status \"ac2b82e2566b72cbef827bd81cdf7d7b00922917ab4eaf47fae07db85ab0fd8c\": rpc error: code = NotFound desc = could not find container \"ac2b82e2566b72cbef827bd81cdf7d7b00922917ab4eaf47fae07db85ab0fd8c\": container with ID starting with ac2b82e2566b72cbef827bd81cdf7d7b00922917ab4eaf47fae07db85ab0fd8c not found: ID does not exist" Nov 03 20:06:37.587955 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:37.587932 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-r24km"] Nov 03 20:06:37.592146 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:37.592126 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-r24km"] Nov 03 20:06:39.212328 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:39.212292 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3" path="/var/lib/kubelet/pods/c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3/volumes" Nov 03 20:06:39.560115 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:39.560044 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" Nov 03 20:06:39.616237 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:39.616204 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg"] Nov 03 20:06:39.616422 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:39.616402 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" podUID="0ed8754d-0475-4c68-8c8c-c2db790f6c89" containerName="maas-api" containerID="cri-o://ff1ea6747bf5683945149a9d2bef3110b1b181383288456ea92becb40508775b" gracePeriod=30 Nov 03 20:06:39.865931 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:39.865907 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" Nov 03 20:06:39.892290 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:39.892261 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed8754d-0475-4c68-8c8c-c2db790f6c89-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed8754d-0475-4c68-8c8c-c2db790f6c89-openshift-service-ca\") pod \"0ed8754d-0475-4c68-8c8c-c2db790f6c89\" (UID: \"0ed8754d-0475-4c68-8c8c-c2db790f6c89\") " Nov 03 20:06:39.892465 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:39.892352 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ed8754d-0475-4c68-8c8c-c2db790f6c89-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ed8754d-0475-4c68-8c8c-c2db790f6c89-maas-api-tls\") pod \"0ed8754d-0475-4c68-8c8c-c2db790f6c89\" (UID: \"0ed8754d-0475-4c68-8c8c-c2db790f6c89\") " Nov 03 20:06:39.892465 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:39.892374 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ed8754d-0475-4c68-8c8c-c2db790f6c89-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ed8754d-0475-4c68-8c8c-c2db790f6c89-metrics-certs\") pod \"0ed8754d-0475-4c68-8c8c-c2db790f6c89\" (UID: \"0ed8754d-0475-4c68-8c8c-c2db790f6c89\") " Nov 03 20:06:39.892465 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:39.892416 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed8754d-0475-4c68-8c8c-c2db790f6c89-kube-api-access-qtsxq\" (UniqueName: \"kubernetes.io/projected/0ed8754d-0475-4c68-8c8c-c2db790f6c89-kube-api-access-qtsxq\") pod \"0ed8754d-0475-4c68-8c8c-c2db790f6c89\" (UID: \"0ed8754d-0475-4c68-8c8c-c2db790f6c89\") " Nov 03 20:06:39.892662 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:39.892636 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed8754d-0475-4c68-8c8c-c2db790f6c89-openshift-service-ca" pod "0ed8754d-0475-4c68-8c8c-c2db790f6c89" (UID: "0ed8754d-0475-4c68-8c8c-c2db790f6c89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:39.894634 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:39.894609 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed8754d-0475-4c68-8c8c-c2db790f6c89-kube-api-access-qtsxq" pod "0ed8754d-0475-4c68-8c8c-c2db790f6c89" (UID: "0ed8754d-0475-4c68-8c8c-c2db790f6c89"). InnerVolumeSpecName "kube-api-access-qtsxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:39.894744 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:39.894672 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ed8754d-0475-4c68-8c8c-c2db790f6c89-metrics-certs" pod "0ed8754d-0475-4c68-8c8c-c2db790f6c89" (UID: "0ed8754d-0475-4c68-8c8c-c2db790f6c89"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:39.894744 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:39.894712 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ed8754d-0475-4c68-8c8c-c2db790f6c89-maas-api-tls" pod "0ed8754d-0475-4c68-8c8c-c2db790f6c89" (UID: "0ed8754d-0475-4c68-8c8c-c2db790f6c89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:39.993705 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:39.993661 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtsxq\" (UniqueName: \"kubernetes.io/projected/0ed8754d-0475-4c68-8c8c-c2db790f6c89-kube-api-access-qtsxq\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:39.993705 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:39.993696 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed8754d-0475-4c68-8c8c-c2db790f6c89-openshift-service-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:39.993931 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:39.993714 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ed8754d-0475-4c68-8c8c-c2db790f6c89-maas-api-tls\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:39.993931 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:39.993728 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ed8754d-0475-4c68-8c8c-c2db790f6c89-metrics-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:40.584970 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:40.584934 4918 generic.go:356] "Generic (PLEG): container finished" podID="0ed8754d-0475-4c68-8c8c-c2db790f6c89" containerID="ff1ea6747bf5683945149a9d2bef3110b1b181383288456ea92becb40508775b" exitCode=0 Nov 03 20:06:40.585439 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:40.584998 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" Nov 03 20:06:40.585439 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:40.585014 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" event={"ID":"0ed8754d-0475-4c68-8c8c-c2db790f6c89","Type":"ContainerDied","Data":"ff1ea6747bf5683945149a9d2bef3110b1b181383288456ea92becb40508775b"} Nov 03 20:06:40.585439 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:40.585086 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg" event={"ID":"0ed8754d-0475-4c68-8c8c-c2db790f6c89","Type":"ContainerDied","Data":"7634cd5fa06473d521df8b7179a78948b558d7038ea85fa15d67bc8c4a101815"} Nov 03 20:06:40.585439 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:40.585120 4918 scope.go:122] "RemoveContainer" containerID="ff1ea6747bf5683945149a9d2bef3110b1b181383288456ea92becb40508775b" Nov 03 20:06:40.594881 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:40.594862 4918 scope.go:122] "RemoveContainer" containerID="ff1ea6747bf5683945149a9d2bef3110b1b181383288456ea92becb40508775b" Nov 03 20:06:40.595131 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:06:40.595110 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1ea6747bf5683945149a9d2bef3110b1b181383288456ea92becb40508775b\": container with ID starting with ff1ea6747bf5683945149a9d2bef3110b1b181383288456ea92becb40508775b not found: ID does not exist" containerID="ff1ea6747bf5683945149a9d2bef3110b1b181383288456ea92becb40508775b" Nov 03 20:06:40.595169 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:40.595140 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1ea6747bf5683945149a9d2bef3110b1b181383288456ea92becb40508775b"} err="failed to get container status \"ff1ea6747bf5683945149a9d2bef3110b1b181383288456ea92becb40508775b\": rpc error: code = NotFound desc = could not find container \"ff1ea6747bf5683945149a9d2bef3110b1b181383288456ea92becb40508775b\": container with ID starting with ff1ea6747bf5683945149a9d2bef3110b1b181383288456ea92becb40508775b not found: ID does not exist" Nov 03 20:06:40.607925 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:40.607897 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg"] Nov 03 20:06:40.612131 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:40.612106 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-sxphg"] Nov 03 20:06:41.212292 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:41.212259 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed8754d-0475-4c68-8c8c-c2db790f6c89" path="/var/lib/kubelet/pods/0ed8754d-0475-4c68-8c8c-c2db790f6c89/volumes" Nov 03 20:06:57.253371 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:57.253345 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:06:57.253903 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:06:57.253882 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:06:58.206597 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:06:58.206560 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:07:28.207376 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:07:28.207335 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:07:48.862039 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:48.862009 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-bzdwr"] Nov 03 20:07:48.862460 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:48.862363 4918 cpu_manager.go:414] "containerMap: removing container" podUID="0ed8754d-0475-4c68-8c8c-c2db790f6c89" containerName="maas-api" Nov 03 20:07:48.862460 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:48.862376 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed8754d-0475-4c68-8c8c-c2db790f6c89" containerName="maas-api" Nov 03 20:07:48.862460 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:48.862401 4918 cpu_manager.go:414] "containerMap: removing container" podUID="c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3" containerName="authorino" Nov 03 20:07:48.862460 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:48.862406 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3" containerName="authorino" Nov 03 20:07:48.862591 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:48.862476 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed8754d-0475-4c68-8c8c-c2db790f6c89" containerName="maas-api" Nov 03 20:07:48.862591 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:48.862488 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ff5b31-5c89-4c2b-bbf7-5b81e68981b3" containerName="authorino" Nov 03 20:07:48.865570 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:48.865546 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:48.870501 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:48.870480 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6vbjn\"" Nov 03 20:07:48.871710 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:48.871686 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 20:07:48.872156 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:48.872080 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 20:07:48.872392 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:48.872170 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 20:07:48.876592 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:48.876567 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-bzdwr"] Nov 03 20:07:48.960522 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:48.960481 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-bzdwr\" (UID: \"fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:48.960706 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:48.960535 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj2hr\" (UniqueName: \"kubernetes.io/projected/fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a-kube-api-access-rj2hr\") pod \"echo-simulator-kserve-64b6f8f96c-bzdwr\" (UID: \"fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:48.960706 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:48.960626 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-bzdwr\" (UID: \"fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:48.960832 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:48.960712 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a-home\") pod \"echo-simulator-kserve-64b6f8f96c-bzdwr\" (UID: \"fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:48.960832 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:48.960761 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-bzdwr\" (UID: \"fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:48.960832 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:48.960799 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-bzdwr\" (UID: \"fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:49.061128 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:49.061094 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-bzdwr\" (UID: \"fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:49.061312 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:49.061140 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-bzdwr\" (UID: \"fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:49.061312 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:49.061195 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj2hr\" (UniqueName: \"kubernetes.io/projected/fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a-kube-api-access-rj2hr\") pod \"echo-simulator-kserve-64b6f8f96c-bzdwr\" (UID: \"fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:49.061312 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:49.061213 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-bzdwr\" (UID: \"fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:49.061312 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:49.061232 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a-home\") pod \"echo-simulator-kserve-64b6f8f96c-bzdwr\" (UID: \"fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:49.061312 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:49.061255 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-bzdwr\" (UID: \"fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:49.061701 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:49.061672 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-bzdwr\" (UID: \"fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:49.061701 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:49.061696 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-bzdwr\" (UID: \"fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:49.061873 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:49.061754 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a-home\") pod \"echo-simulator-kserve-64b6f8f96c-bzdwr\" (UID: \"fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:49.063451 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:49.063421 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-bzdwr\" (UID: \"fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:49.063668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:49.063649 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-bzdwr\" (UID: \"fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:49.073960 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:49.073937 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj2hr\" (UniqueName: \"kubernetes.io/projected/fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a-kube-api-access-rj2hr\") pod \"echo-simulator-kserve-64b6f8f96c-bzdwr\" (UID: \"fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:49.180509 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:49.180418 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:49.456961 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:49.456928 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-bzdwr"] Nov 03 20:07:49.459712 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:07:49.459682 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa956d00_6eba_4ef7_bbb8_d63ed2dbdb9a.slice/crio-8ed26d95063cac13065e713322931313d52fa40c50d296b8eda84313c2886a3a WatchSource:0}: Error finding container 8ed26d95063cac13065e713322931313d52fa40c50d296b8eda84313c2886a3a: Status 404 returned error can't find the container with id 8ed26d95063cac13065e713322931313d52fa40c50d296b8eda84313c2886a3a Nov 03 20:07:49.872383 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:49.872345 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" event={"ID":"fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a","Type":"ContainerStarted","Data":"8ed26d95063cac13065e713322931313d52fa40c50d296b8eda84313c2886a3a"} Nov 03 20:07:50.183294 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.183222 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk"] Nov 03 20:07:50.186452 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.186433 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:50.191428 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.191384 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 20:07:50.196700 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.196634 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk"] Nov 03 20:07:50.270687 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.270652 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9493241-95b1-4b6d-9290-c677b2a55000-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk\" (UID: \"c9493241-95b1-4b6d-9290-c677b2a55000\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:50.270854 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.270705 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9493241-95b1-4b6d-9290-c677b2a55000-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk\" (UID: \"c9493241-95b1-4b6d-9290-c677b2a55000\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:50.270903 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.270870 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9493241-95b1-4b6d-9290-c677b2a55000-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk\" (UID: \"c9493241-95b1-4b6d-9290-c677b2a55000\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:50.270950 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.270913 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmc7s\" (UniqueName: \"kubernetes.io/projected/c9493241-95b1-4b6d-9290-c677b2a55000-kube-api-access-fmc7s\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk\" (UID: \"c9493241-95b1-4b6d-9290-c677b2a55000\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:50.270986 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.270963 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9493241-95b1-4b6d-9290-c677b2a55000-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk\" (UID: \"c9493241-95b1-4b6d-9290-c677b2a55000\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:50.271018 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.270996 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9493241-95b1-4b6d-9290-c677b2a55000-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk\" (UID: \"c9493241-95b1-4b6d-9290-c677b2a55000\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:50.371806 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.371771 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9493241-95b1-4b6d-9290-c677b2a55000-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk\" (UID: \"c9493241-95b1-4b6d-9290-c677b2a55000\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:50.371971 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.371840 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9493241-95b1-4b6d-9290-c677b2a55000-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk\" (UID: \"c9493241-95b1-4b6d-9290-c677b2a55000\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:50.371971 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.371895 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9493241-95b1-4b6d-9290-c677b2a55000-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk\" (UID: \"c9493241-95b1-4b6d-9290-c677b2a55000\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:50.371971 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.371924 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmc7s\" (UniqueName: \"kubernetes.io/projected/c9493241-95b1-4b6d-9290-c677b2a55000-kube-api-access-fmc7s\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk\" (UID: \"c9493241-95b1-4b6d-9290-c677b2a55000\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:50.371971 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.371946 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9493241-95b1-4b6d-9290-c677b2a55000-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk\" (UID: \"c9493241-95b1-4b6d-9290-c677b2a55000\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:50.372143 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.371982 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9493241-95b1-4b6d-9290-c677b2a55000-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk\" (UID: \"c9493241-95b1-4b6d-9290-c677b2a55000\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:50.372238 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.372214 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9493241-95b1-4b6d-9290-c677b2a55000-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk\" (UID: \"c9493241-95b1-4b6d-9290-c677b2a55000\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:50.372677 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.372651 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9493241-95b1-4b6d-9290-c677b2a55000-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk\" (UID: \"c9493241-95b1-4b6d-9290-c677b2a55000\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:50.372790 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.372770 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9493241-95b1-4b6d-9290-c677b2a55000-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk\" (UID: \"c9493241-95b1-4b6d-9290-c677b2a55000\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:50.374824 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.374801 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9493241-95b1-4b6d-9290-c677b2a55000-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk\" (UID: \"c9493241-95b1-4b6d-9290-c677b2a55000\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:50.374935 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.374904 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9493241-95b1-4b6d-9290-c677b2a55000-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk\" (UID: \"c9493241-95b1-4b6d-9290-c677b2a55000\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:50.387837 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.387793 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmc7s\" (UniqueName: \"kubernetes.io/projected/c9493241-95b1-4b6d-9290-c677b2a55000-kube-api-access-fmc7s\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk\" (UID: \"c9493241-95b1-4b6d-9290-c677b2a55000\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:50.503493 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.503409 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:50.788335 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.788306 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk"] Nov 03 20:07:50.790723 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:07:50.790693 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9493241_95b1_4b6d_9290_c677b2a55000.slice/crio-02edbd2fa46369ea195bf3f1c4d9da7e959ae786c49105ca45c8103531c8eba2 WatchSource:0}: Error finding container 02edbd2fa46369ea195bf3f1c4d9da7e959ae786c49105ca45c8103531c8eba2: Status 404 returned error can't find the container with id 02edbd2fa46369ea195bf3f1c4d9da7e959ae786c49105ca45c8103531c8eba2 Nov 03 20:07:50.878371 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:50.878338 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" event={"ID":"c9493241-95b1-4b6d-9290-c677b2a55000","Type":"ContainerStarted","Data":"02edbd2fa46369ea195bf3f1c4d9da7e959ae786c49105ca45c8103531c8eba2"} Nov 03 20:07:51.304116 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.304082 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9"] Nov 03 20:07:51.309613 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.309412 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:51.314095 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.314026 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 20:07:51.320067 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.320043 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9"] Nov 03 20:07:51.382675 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.382624 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ad604d4-e127-4bd4-9d38-6f8b3987b313-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9\" (UID: \"8ad604d4-e127-4bd4-9d38-6f8b3987b313\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:51.382675 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.382676 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ad604d4-e127-4bd4-9d38-6f8b3987b313-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9\" (UID: \"8ad604d4-e127-4bd4-9d38-6f8b3987b313\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:51.382935 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.382711 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ad604d4-e127-4bd4-9d38-6f8b3987b313-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9\" (UID: \"8ad604d4-e127-4bd4-9d38-6f8b3987b313\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:51.382935 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.382747 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6cpc\" (UniqueName: \"kubernetes.io/projected/8ad604d4-e127-4bd4-9d38-6f8b3987b313-kube-api-access-s6cpc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9\" (UID: \"8ad604d4-e127-4bd4-9d38-6f8b3987b313\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:51.382935 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.382813 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ad604d4-e127-4bd4-9d38-6f8b3987b313-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9\" (UID: \"8ad604d4-e127-4bd4-9d38-6f8b3987b313\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:51.382935 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.382856 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ad604d4-e127-4bd4-9d38-6f8b3987b313-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9\" (UID: \"8ad604d4-e127-4bd4-9d38-6f8b3987b313\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:51.483482 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.483437 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6cpc\" (UniqueName: \"kubernetes.io/projected/8ad604d4-e127-4bd4-9d38-6f8b3987b313-kube-api-access-s6cpc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9\" (UID: \"8ad604d4-e127-4bd4-9d38-6f8b3987b313\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:51.483711 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.483535 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ad604d4-e127-4bd4-9d38-6f8b3987b313-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9\" (UID: \"8ad604d4-e127-4bd4-9d38-6f8b3987b313\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:51.483711 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.483582 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ad604d4-e127-4bd4-9d38-6f8b3987b313-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9\" (UID: \"8ad604d4-e127-4bd4-9d38-6f8b3987b313\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:51.483711 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.483645 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ad604d4-e127-4bd4-9d38-6f8b3987b313-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9\" (UID: \"8ad604d4-e127-4bd4-9d38-6f8b3987b313\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:51.483711 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.483676 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ad604d4-e127-4bd4-9d38-6f8b3987b313-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9\" (UID: \"8ad604d4-e127-4bd4-9d38-6f8b3987b313\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:51.483711 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.483702 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ad604d4-e127-4bd4-9d38-6f8b3987b313-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9\" (UID: \"8ad604d4-e127-4bd4-9d38-6f8b3987b313\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:51.484318 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.484078 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ad604d4-e127-4bd4-9d38-6f8b3987b313-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9\" (UID: \"8ad604d4-e127-4bd4-9d38-6f8b3987b313\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:51.484318 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.484204 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ad604d4-e127-4bd4-9d38-6f8b3987b313-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9\" (UID: \"8ad604d4-e127-4bd4-9d38-6f8b3987b313\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:51.484318 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.484267 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ad604d4-e127-4bd4-9d38-6f8b3987b313-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9\" (UID: \"8ad604d4-e127-4bd4-9d38-6f8b3987b313\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:51.486424 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.486398 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ad604d4-e127-4bd4-9d38-6f8b3987b313-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9\" (UID: \"8ad604d4-e127-4bd4-9d38-6f8b3987b313\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:51.487077 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.487053 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ad604d4-e127-4bd4-9d38-6f8b3987b313-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9\" (UID: \"8ad604d4-e127-4bd4-9d38-6f8b3987b313\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:51.496325 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.496267 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6cpc\" (UniqueName: \"kubernetes.io/projected/8ad604d4-e127-4bd4-9d38-6f8b3987b313-kube-api-access-s6cpc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9\" (UID: \"8ad604d4-e127-4bd4-9d38-6f8b3987b313\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:51.641061 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.641024 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:51.889317 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.889281 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" event={"ID":"fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a","Type":"ContainerStarted","Data":"bfdb6f38320cfe6c93cbc14f3751ee3b408cd23508d0fca6f7cc7674515a7527"} Nov 03 20:07:51.910870 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.910807 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" podStartSLOduration=2.123957849 podStartE2EDuration="3.910786522s" totalImagesPullingTime="1.786828673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:51.909446865 +0000 UTC m=+654.841314391" watchObservedRunningTime="2026-11-03 20:07:51.910786522 +0000 UTC m=+654.842654034" Nov 03 20:07:51.930240 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:51.930209 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9"] Nov 03 20:07:51.932864 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:07:51.932829 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ad604d4_e127_4bd4_9d38_6f8b3987b313.slice/crio-119bb7d21feba616939e580761f4ea86dfccdfc93f1cde49a7ccd0b6fe537487 WatchSource:0}: Error finding container 119bb7d21feba616939e580761f4ea86dfccdfc93f1cde49a7ccd0b6fe537487: Status 404 returned error can't find the container with id 119bb7d21feba616939e580761f4ea86dfccdfc93f1cde49a7ccd0b6fe537487 Nov 03 20:07:52.181288 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.181218 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:52.474518 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.474436 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4"] Nov 03 20:07:52.477941 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.477917 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:52.482323 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.482300 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 20:07:52.487788 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.487757 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4"] Nov 03 20:07:52.593690 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.593642 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d51a14a7-ad15-477a-8eb6-83b5fc798851-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4\" (UID: \"d51a14a7-ad15-477a-8eb6-83b5fc798851\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:52.593867 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.593698 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d51a14a7-ad15-477a-8eb6-83b5fc798851-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4\" (UID: \"d51a14a7-ad15-477a-8eb6-83b5fc798851\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:52.593867 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.593774 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d51a14a7-ad15-477a-8eb6-83b5fc798851-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4\" (UID: \"d51a14a7-ad15-477a-8eb6-83b5fc798851\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:52.593867 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.593827 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d51a14a7-ad15-477a-8eb6-83b5fc798851-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4\" (UID: \"d51a14a7-ad15-477a-8eb6-83b5fc798851\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:52.594032 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.593935 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st8hh\" (UniqueName: \"kubernetes.io/projected/d51a14a7-ad15-477a-8eb6-83b5fc798851-kube-api-access-st8hh\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4\" (UID: \"d51a14a7-ad15-477a-8eb6-83b5fc798851\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:52.594032 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.593973 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d51a14a7-ad15-477a-8eb6-83b5fc798851-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4\" (UID: \"d51a14a7-ad15-477a-8eb6-83b5fc798851\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:52.695150 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.695115 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d51a14a7-ad15-477a-8eb6-83b5fc798851-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4\" (UID: \"d51a14a7-ad15-477a-8eb6-83b5fc798851\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:52.695150 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.695155 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d51a14a7-ad15-477a-8eb6-83b5fc798851-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4\" (UID: \"d51a14a7-ad15-477a-8eb6-83b5fc798851\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:52.695465 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.695173 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d51a14a7-ad15-477a-8eb6-83b5fc798851-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4\" (UID: \"d51a14a7-ad15-477a-8eb6-83b5fc798851\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:52.695465 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.695223 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d51a14a7-ad15-477a-8eb6-83b5fc798851-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4\" (UID: \"d51a14a7-ad15-477a-8eb6-83b5fc798851\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:52.695465 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.695272 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st8hh\" (UniqueName: \"kubernetes.io/projected/d51a14a7-ad15-477a-8eb6-83b5fc798851-kube-api-access-st8hh\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4\" (UID: \"d51a14a7-ad15-477a-8eb6-83b5fc798851\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:52.695465 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.695328 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d51a14a7-ad15-477a-8eb6-83b5fc798851-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4\" (UID: \"d51a14a7-ad15-477a-8eb6-83b5fc798851\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:52.695662 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.695575 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d51a14a7-ad15-477a-8eb6-83b5fc798851-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4\" (UID: \"d51a14a7-ad15-477a-8eb6-83b5fc798851\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:52.695711 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.695657 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d51a14a7-ad15-477a-8eb6-83b5fc798851-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4\" (UID: \"d51a14a7-ad15-477a-8eb6-83b5fc798851\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:52.695746 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.695711 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d51a14a7-ad15-477a-8eb6-83b5fc798851-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4\" (UID: \"d51a14a7-ad15-477a-8eb6-83b5fc798851\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:52.697582 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.697562 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d51a14a7-ad15-477a-8eb6-83b5fc798851-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4\" (UID: \"d51a14a7-ad15-477a-8eb6-83b5fc798851\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:52.697782 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.697763 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d51a14a7-ad15-477a-8eb6-83b5fc798851-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4\" (UID: \"d51a14a7-ad15-477a-8eb6-83b5fc798851\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:52.707373 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.707354 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st8hh\" (UniqueName: \"kubernetes.io/projected/d51a14a7-ad15-477a-8eb6-83b5fc798851-kube-api-access-st8hh\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4\" (UID: \"d51a14a7-ad15-477a-8eb6-83b5fc798851\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:52.791291 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.791206 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:52.896441 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.896407 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" event={"ID":"c9493241-95b1-4b6d-9290-c677b2a55000","Type":"ContainerStarted","Data":"2d154fd4cf12d5ab5c153b41f7332b4fd66c3f2b469ef0ec7f425342c2c29d9b"} Nov 03 20:07:52.898052 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.898025 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" event={"ID":"8ad604d4-e127-4bd4-9d38-6f8b3987b313","Type":"ContainerStarted","Data":"152662ef9195b14ea4759603b08d09139a6184823f0120d7ce5c7e15e9856dfe"} Nov 03 20:07:52.898161 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.898064 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" event={"ID":"8ad604d4-e127-4bd4-9d38-6f8b3987b313","Type":"ContainerStarted","Data":"119bb7d21feba616939e580761f4ea86dfccdfc93f1cde49a7ccd0b6fe537487"} Nov 03 20:07:52.917365 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.917318 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" podStartSLOduration=1.6324667320000001 podStartE2EDuration="2.917304254s" totalImagesPullingTime="1.284837522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:52.916407003 +0000 UTC m=+655.848274518" watchObservedRunningTime="2026-11-03 20:07:52.917304254 +0000 UTC m=+655.849171765" Nov 03 20:07:52.936598 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:52.936550 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podStartSLOduration=1.590824411 podStartE2EDuration="1.936539152s" totalImagesPullingTime="345.714741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:52.935591332 +0000 UTC m=+655.867458843" watchObservedRunningTime="2026-11-03 20:07:52.936539152 +0000 UTC m=+655.868406662" Nov 03 20:07:53.063947 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.063827 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4"] Nov 03 20:07:53.066039 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:07:53.066009 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd51a14a7_ad15_477a_8eb6_83b5fc798851.slice/crio-93b74a1a02123acfb5d4340848067e344e116f1ea2b49f0b5b1d452ef8c5278a WatchSource:0}: Error finding container 93b74a1a02123acfb5d4340848067e344e116f1ea2b49f0b5b1d452ef8c5278a: Status 404 returned error can't find the container with id 93b74a1a02123acfb5d4340848067e344e116f1ea2b49f0b5b1d452ef8c5278a Nov 03 20:07:53.467327 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.467301 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6"] Nov 03 20:07:53.470548 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.470527 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:53.477836 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.477802 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 20:07:53.482620 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.482592 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6"] Nov 03 20:07:53.502507 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.502470 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1a63d7a-3f9e-46dc-8e39-144eb440c4b5-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgcx6\" (UID: \"e1a63d7a-3f9e-46dc-8e39-144eb440c4b5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:53.502684 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.502648 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1a63d7a-3f9e-46dc-8e39-144eb440c4b5-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgcx6\" (UID: \"e1a63d7a-3f9e-46dc-8e39-144eb440c4b5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:53.502786 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.502719 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1a63d7a-3f9e-46dc-8e39-144eb440c4b5-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgcx6\" (UID: \"e1a63d7a-3f9e-46dc-8e39-144eb440c4b5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:53.502786 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.502766 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a63d7a-3f9e-46dc-8e39-144eb440c4b5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgcx6\" (UID: \"e1a63d7a-3f9e-46dc-8e39-144eb440c4b5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:53.502879 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.502795 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1a63d7a-3f9e-46dc-8e39-144eb440c4b5-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgcx6\" (UID: \"e1a63d7a-3f9e-46dc-8e39-144eb440c4b5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:53.502879 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.502844 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjtr\" (UniqueName: \"kubernetes.io/projected/e1a63d7a-3f9e-46dc-8e39-144eb440c4b5-kube-api-access-6xjtr\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgcx6\" (UID: \"e1a63d7a-3f9e-46dc-8e39-144eb440c4b5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:53.504080 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.504059 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:07:53.604337 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.604291 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1a63d7a-3f9e-46dc-8e39-144eb440c4b5-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgcx6\" (UID: \"e1a63d7a-3f9e-46dc-8e39-144eb440c4b5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:53.604536 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.604362 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1a63d7a-3f9e-46dc-8e39-144eb440c4b5-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgcx6\" (UID: \"e1a63d7a-3f9e-46dc-8e39-144eb440c4b5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:53.604536 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.604391 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1a63d7a-3f9e-46dc-8e39-144eb440c4b5-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgcx6\" (UID: \"e1a63d7a-3f9e-46dc-8e39-144eb440c4b5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:53.604536 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.604425 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a63d7a-3f9e-46dc-8e39-144eb440c4b5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgcx6\" (UID: \"e1a63d7a-3f9e-46dc-8e39-144eb440c4b5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:53.604698 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.604552 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1a63d7a-3f9e-46dc-8e39-144eb440c4b5-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgcx6\" (UID: \"e1a63d7a-3f9e-46dc-8e39-144eb440c4b5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:53.604698 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.604624 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjtr\" (UniqueName: \"kubernetes.io/projected/e1a63d7a-3f9e-46dc-8e39-144eb440c4b5-kube-api-access-6xjtr\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgcx6\" (UID: \"e1a63d7a-3f9e-46dc-8e39-144eb440c4b5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:53.604809 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.604742 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1a63d7a-3f9e-46dc-8e39-144eb440c4b5-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgcx6\" (UID: \"e1a63d7a-3f9e-46dc-8e39-144eb440c4b5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:53.604809 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.604771 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1a63d7a-3f9e-46dc-8e39-144eb440c4b5-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgcx6\" (UID: \"e1a63d7a-3f9e-46dc-8e39-144eb440c4b5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:53.604890 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.604814 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a63d7a-3f9e-46dc-8e39-144eb440c4b5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgcx6\" (UID: \"e1a63d7a-3f9e-46dc-8e39-144eb440c4b5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:53.606848 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.606819 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1a63d7a-3f9e-46dc-8e39-144eb440c4b5-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgcx6\" (UID: \"e1a63d7a-3f9e-46dc-8e39-144eb440c4b5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:53.606980 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.606962 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1a63d7a-3f9e-46dc-8e39-144eb440c4b5-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgcx6\" (UID: \"e1a63d7a-3f9e-46dc-8e39-144eb440c4b5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:53.617209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.617167 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjtr\" (UniqueName: \"kubernetes.io/projected/e1a63d7a-3f9e-46dc-8e39-144eb440c4b5-kube-api-access-6xjtr\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgcx6\" (UID: \"e1a63d7a-3f9e-46dc-8e39-144eb440c4b5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:53.641922 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.641898 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:07:53.787293 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.787211 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:53.904125 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.904089 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" event={"ID":"d51a14a7-ad15-477a-8eb6-83b5fc798851","Type":"ContainerStarted","Data":"524940be5d86083a525875d705a0f7da8559d2c54a3c93ab7d92dab0fb8b9404"} Nov 03 20:07:53.904638 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.904140 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" event={"ID":"d51a14a7-ad15-477a-8eb6-83b5fc798851","Type":"ContainerStarted","Data":"93b74a1a02123acfb5d4340848067e344e116f1ea2b49f0b5b1d452ef8c5278a"} Nov 03 20:07:53.925019 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:53.924971 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" podStartSLOduration=1.588087729 podStartE2EDuration="1.924957668s" totalImagesPullingTime="336.869939ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:53.923943575 +0000 UTC m=+656.855811079" watchObservedRunningTime="2026-11-03 20:07:53.924957668 +0000 UTC m=+656.856825178" Nov 03 20:07:54.063836 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:54.063811 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6"] Nov 03 20:07:54.065519 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:07:54.065491 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a63d7a_3f9e_46dc_8e39_144eb440c4b5.slice/crio-d582d3789c1ea1af9de9ec8f176e90761319087dab335cdca7bce9394efae424 WatchSource:0}: Error finding container d582d3789c1ea1af9de9ec8f176e90761319087dab335cdca7bce9394efae424: Status 404 returned error can't find the container with id d582d3789c1ea1af9de9ec8f176e90761319087dab335cdca7bce9394efae424 Nov 03 20:07:54.791663 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:54.791581 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:07:54.910301 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:54.910255 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" event={"ID":"e1a63d7a-3f9e-46dc-8e39-144eb440c4b5","Type":"ContainerStarted","Data":"b422867055876dedcb1803a7990cf235ceecbfef9928afd5f47e5ab9c02d579f"} Nov 03 20:07:54.910720 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:54.910322 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" event={"ID":"e1a63d7a-3f9e-46dc-8e39-144eb440c4b5","Type":"ContainerStarted","Data":"d582d3789c1ea1af9de9ec8f176e90761319087dab335cdca7bce9394efae424"} Nov 03 20:07:54.931760 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:54.931712 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" podStartSLOduration=1.612481807 podStartE2EDuration="1.931697184s" totalImagesPullingTime="319.215377ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:54.930444818 +0000 UTC m=+657.862312333" watchObservedRunningTime="2026-11-03 20:07:54.931697184 +0000 UTC m=+657.863564688" Nov 03 20:07:55.787950 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:55.787909 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:07:56.103482 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.103440 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr"] Nov 03 20:07:56.106934 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.106915 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:56.112519 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.112497 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 20:07:56.116855 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.116829 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr"] Nov 03 20:07:56.229090 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.229058 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a102a4-f391-4d65-8a93-324a300363cc-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7qnhr\" (UID: \"34a102a4-f391-4d65-8a93-324a300363cc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:56.229090 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.229094 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34a102a4-f391-4d65-8a93-324a300363cc-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7qnhr\" (UID: \"34a102a4-f391-4d65-8a93-324a300363cc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:56.229388 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.229118 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a102a4-f391-4d65-8a93-324a300363cc-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7qnhr\" (UID: \"34a102a4-f391-4d65-8a93-324a300363cc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:56.229388 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.229210 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a102a4-f391-4d65-8a93-324a300363cc-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7qnhr\" (UID: \"34a102a4-f391-4d65-8a93-324a300363cc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:56.229388 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.229255 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx2bn\" (UniqueName: \"kubernetes.io/projected/34a102a4-f391-4d65-8a93-324a300363cc-kube-api-access-hx2bn\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7qnhr\" (UID: \"34a102a4-f391-4d65-8a93-324a300363cc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:56.229388 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.229314 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a102a4-f391-4d65-8a93-324a300363cc-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7qnhr\" (UID: \"34a102a4-f391-4d65-8a93-324a300363cc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:56.330457 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.330418 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a102a4-f391-4d65-8a93-324a300363cc-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7qnhr\" (UID: \"34a102a4-f391-4d65-8a93-324a300363cc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:56.330666 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.330465 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34a102a4-f391-4d65-8a93-324a300363cc-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7qnhr\" (UID: \"34a102a4-f391-4d65-8a93-324a300363cc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:56.330731 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.330692 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a102a4-f391-4d65-8a93-324a300363cc-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7qnhr\" (UID: \"34a102a4-f391-4d65-8a93-324a300363cc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:56.330798 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.330733 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a102a4-f391-4d65-8a93-324a300363cc-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7qnhr\" (UID: \"34a102a4-f391-4d65-8a93-324a300363cc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:56.330798 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.330771 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx2bn\" (UniqueName: \"kubernetes.io/projected/34a102a4-f391-4d65-8a93-324a300363cc-kube-api-access-hx2bn\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7qnhr\" (UID: \"34a102a4-f391-4d65-8a93-324a300363cc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:56.330897 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.330860 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a102a4-f391-4d65-8a93-324a300363cc-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7qnhr\" (UID: \"34a102a4-f391-4d65-8a93-324a300363cc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:56.331270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.331238 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a102a4-f391-4d65-8a93-324a300363cc-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7qnhr\" (UID: \"34a102a4-f391-4d65-8a93-324a300363cc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:56.331394 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.331353 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34a102a4-f391-4d65-8a93-324a300363cc-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7qnhr\" (UID: \"34a102a4-f391-4d65-8a93-324a300363cc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:56.331458 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.331405 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a102a4-f391-4d65-8a93-324a300363cc-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7qnhr\" (UID: \"34a102a4-f391-4d65-8a93-324a300363cc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:56.332826 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.332809 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a102a4-f391-4d65-8a93-324a300363cc-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7qnhr\" (UID: \"34a102a4-f391-4d65-8a93-324a300363cc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:56.333220 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.333200 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a102a4-f391-4d65-8a93-324a300363cc-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7qnhr\" (UID: \"34a102a4-f391-4d65-8a93-324a300363cc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:56.343676 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.343644 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx2bn\" (UniqueName: \"kubernetes.io/projected/34a102a4-f391-4d65-8a93-324a300363cc-kube-api-access-hx2bn\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7qnhr\" (UID: \"34a102a4-f391-4d65-8a93-324a300363cc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:56.427647 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.427545 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:56.707447 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.707414 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr"] Nov 03 20:07:56.710245 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:07:56.710217 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a102a4_f391_4d65_8a93_324a300363cc.slice/crio-04e52cec05ad49a2d337d30106ddd6351b37f4309a2e25a957473d89cb31f372 WatchSource:0}: Error finding container 04e52cec05ad49a2d337d30106ddd6351b37f4309a2e25a957473d89cb31f372: Status 404 returned error can't find the container with id 04e52cec05ad49a2d337d30106ddd6351b37f4309a2e25a957473d89cb31f372 Nov 03 20:07:56.919952 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:56.919912 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" event={"ID":"34a102a4-f391-4d65-8a93-324a300363cc","Type":"ContainerStarted","Data":"04e52cec05ad49a2d337d30106ddd6351b37f4309a2e25a957473d89cb31f372"} Nov 03 20:07:57.038886 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.038809 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m"] Nov 03 20:07:57.042620 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.042595 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:57.046838 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.046809 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 20:07:57.061254 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.061192 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m"] Nov 03 20:07:57.139537 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.139510 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/190167d8-3e24-427f-86d2-a53d510603ed-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hn84m\" (UID: \"190167d8-3e24-427f-86d2-a53d510603ed\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:57.139905 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.139559 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/190167d8-3e24-427f-86d2-a53d510603ed-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hn84m\" (UID: \"190167d8-3e24-427f-86d2-a53d510603ed\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:57.139905 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.139649 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/190167d8-3e24-427f-86d2-a53d510603ed-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hn84m\" (UID: \"190167d8-3e24-427f-86d2-a53d510603ed\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:57.139905 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.139756 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190167d8-3e24-427f-86d2-a53d510603ed-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hn84m\" (UID: \"190167d8-3e24-427f-86d2-a53d510603ed\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:57.139905 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.139785 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190167d8-3e24-427f-86d2-a53d510603ed-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hn84m\" (UID: \"190167d8-3e24-427f-86d2-a53d510603ed\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:57.139905 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.139843 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22qvp\" (UniqueName: \"kubernetes.io/projected/190167d8-3e24-427f-86d2-a53d510603ed-kube-api-access-22qvp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hn84m\" (UID: \"190167d8-3e24-427f-86d2-a53d510603ed\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:57.240498 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.240466 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/190167d8-3e24-427f-86d2-a53d510603ed-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hn84m\" (UID: \"190167d8-3e24-427f-86d2-a53d510603ed\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:57.240668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.240529 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190167d8-3e24-427f-86d2-a53d510603ed-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hn84m\" (UID: \"190167d8-3e24-427f-86d2-a53d510603ed\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:57.240668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.240638 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190167d8-3e24-427f-86d2-a53d510603ed-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hn84m\" (UID: \"190167d8-3e24-427f-86d2-a53d510603ed\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:57.240766 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.240713 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22qvp\" (UniqueName: \"kubernetes.io/projected/190167d8-3e24-427f-86d2-a53d510603ed-kube-api-access-22qvp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hn84m\" (UID: \"190167d8-3e24-427f-86d2-a53d510603ed\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:57.240766 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.240757 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/190167d8-3e24-427f-86d2-a53d510603ed-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hn84m\" (UID: \"190167d8-3e24-427f-86d2-a53d510603ed\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:57.240860 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.240822 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/190167d8-3e24-427f-86d2-a53d510603ed-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hn84m\" (UID: \"190167d8-3e24-427f-86d2-a53d510603ed\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:57.240918 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.240861 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190167d8-3e24-427f-86d2-a53d510603ed-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hn84m\" (UID: \"190167d8-3e24-427f-86d2-a53d510603ed\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:57.241006 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.240984 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190167d8-3e24-427f-86d2-a53d510603ed-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hn84m\" (UID: \"190167d8-3e24-427f-86d2-a53d510603ed\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:57.241151 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.241135 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/190167d8-3e24-427f-86d2-a53d510603ed-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hn84m\" (UID: \"190167d8-3e24-427f-86d2-a53d510603ed\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:57.242881 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.242865 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/190167d8-3e24-427f-86d2-a53d510603ed-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hn84m\" (UID: \"190167d8-3e24-427f-86d2-a53d510603ed\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:57.243375 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.243358 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 20:07:57.252936 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.252906 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/190167d8-3e24-427f-86d2-a53d510603ed-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hn84m\" (UID: \"190167d8-3e24-427f-86d2-a53d510603ed\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:57.253096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.253067 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22qvp\" (UniqueName: \"kubernetes.io/projected/190167d8-3e24-427f-86d2-a53d510603ed-kube-api-access-22qvp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hn84m\" (UID: \"190167d8-3e24-427f-86d2-a53d510603ed\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:57.358125 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.358093 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:57.638228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.638198 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m"] Nov 03 20:07:57.640095 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:07:57.640065 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190167d8_3e24_427f_86d2_a53d510603ed.slice/crio-5eea74fecf552367b693f901c3be54ccc71aff487036ab8b3afc6a3e0cb1d636 WatchSource:0}: Error finding container 5eea74fecf552367b693f901c3be54ccc71aff487036ab8b3afc6a3e0cb1d636: Status 404 returned error can't find the container with id 5eea74fecf552367b693f901c3be54ccc71aff487036ab8b3afc6a3e0cb1d636 Nov 03 20:07:57.926215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.926104 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" event={"ID":"34a102a4-f391-4d65-8a93-324a300363cc","Type":"ContainerStarted","Data":"0f6ee23f3a2f172f1bb22da92dd0b9f2691462790ed1b8d4cd82fe5c60567785"} Nov 03 20:07:57.927303 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.927279 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" event={"ID":"190167d8-3e24-427f-86d2-a53d510603ed","Type":"ContainerStarted","Data":"5eea74fecf552367b693f901c3be54ccc71aff487036ab8b3afc6a3e0cb1d636"} Nov 03 20:07:57.947414 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:57.947361 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" podStartSLOduration=1.58894592 podStartE2EDuration="1.947342363s" totalImagesPullingTime="358.396443ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:57.945991144 +0000 UTC m=+660.877858677" watchObservedRunningTime="2026-11-03 20:07:57.947342363 +0000 UTC m=+660.879209874" Nov 03 20:07:58.206267 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:07:58.206162 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:07:58.428430 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:58.428398 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:07:58.933246 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:58.933208 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" event={"ID":"190167d8-3e24-427f-86d2-a53d510603ed","Type":"ContainerStarted","Data":"2c0fe9d63ad1cddcce859a8fc74ad170f80a8ac66ca151ad4f32affa044af38f"} Nov 03 20:07:58.957796 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:58.957742 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" podStartSLOduration=1.6185173370000001 podStartE2EDuration="1.957727985s" totalImagesPullingTime="339.210648ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:58.955973491 +0000 UTC m=+661.887841002" watchObservedRunningTime="2026-11-03 20:07:58.957727985 +0000 UTC m=+661.889595492" Nov 03 20:07:59.180911 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:59.180862 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:59.193524 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:59.193457 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:07:59.358445 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:59.358404 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:07:59.949486 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:07:59.949458 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" Nov 03 20:08:00.504115 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:00.504075 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:08:00.516457 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:00.516429 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:08:00.954509 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:00.954477 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" Nov 03 20:08:01.641373 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:01.641335 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:08:01.653647 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:01.653620 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:08:01.962400 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:01.962310 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:08:02.792403 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:02.792364 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:08:02.804827 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:02.804803 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:08:02.964074 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:02.964045 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4" Nov 03 20:08:03.787529 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:03.787485 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:08:03.799917 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:03.799890 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:08:03.969627 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:03.969594 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgcx6" Nov 03 20:08:06.428752 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:06.428714 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:08:06.441128 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:06.441100 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:08:06.984217 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:06.984173 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" Nov 03 20:08:07.359113 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:07.359080 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:08:07.371755 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:07.371723 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:08:07.989556 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:07.989475 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hn84m" Nov 03 20:08:24.868356 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:24.868303 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr"] Nov 03 20:08:24.875838 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:24.875811 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:24.885033 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:24.885007 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 20:08:24.885203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:24.885102 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gwd2g\"" Nov 03 20:08:24.887292 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:24.887264 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr"] Nov 03 20:08:24.992073 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:24.992032 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:24.992268 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:24.992088 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:24.992268 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:24.992107 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:24.992268 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:24.992231 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:24.992458 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:24.992270 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:24.992458 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:24.992381 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da26a005-8c1e-46a4-8a17-93119a8a735d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:24.992458 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:24.992413 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fnsq\" (UniqueName: \"kubernetes.io/projected/da26a005-8c1e-46a4-8a17-93119a8a735d-kube-api-access-2fnsq\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:24.992458 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:24.992448 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:24.992618 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:24.992523 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:25.093034 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.092998 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:25.093214 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.093048 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:25.093214 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.093068 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:25.093214 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.093093 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:25.093214 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.093124 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:25.093431 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.093315 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da26a005-8c1e-46a4-8a17-93119a8a735d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:25.093431 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.093364 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fnsq\" (UniqueName: \"kubernetes.io/projected/da26a005-8c1e-46a4-8a17-93119a8a735d-kube-api-access-2fnsq\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:25.093431 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.093397 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:25.093576 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.093435 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:25.093576 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.093452 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:25.093669 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.093653 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:25.093824 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.093799 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:25.093899 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.093883 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:25.094063 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.094039 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da26a005-8c1e-46a4-8a17-93119a8a735d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:25.095560 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.095528 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:25.095744 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.095719 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:25.106667 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.106643 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fnsq\" (UniqueName: \"kubernetes.io/projected/da26a005-8c1e-46a4-8a17-93119a8a735d-kube-api-access-2fnsq\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:25.106810 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.106786 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-zjhwr\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:25.189147 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.189059 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:25.472739 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.472713 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr"] Nov 03 20:08:25.478875 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:25.478856 4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:08:26.059730 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:26.059694 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" event={"ID":"da26a005-8c1e-46a4-8a17-93119a8a735d","Type":"ContainerStarted","Data":"80bed84f133bcda43ee467d76850e603878824cf2fc2c9011f0f34c371e2b10e"} Nov 03 20:08:28.223001 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:08:28.222956 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:08:28.921239 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:28.921204 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:08:28.921364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:28.921285 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:08:28.921364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:28.921328 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:08:29.076746 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:29.076707 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" event={"ID":"da26a005-8c1e-46a4-8a17-93119a8a735d","Type":"ContainerStarted","Data":"ef21e567cda94fc66e98c4cfc1fc93a265404e8091928a8687dab1c20799410e"} Nov 03 20:08:29.103716 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:29.103661 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" podStartSLOduration=1.662370267 podStartE2EDuration="5.103647468s" totalImagesPullingTime="3.441277201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:29.101574338 +0000 UTC m=+692.033441842" watchObservedRunningTime="2026-11-03 20:08:29.103647468 +0000 UTC m=+692.035514979" Nov 03 20:08:29.189695 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:29.189612 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:29.191030 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:29.190999 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zjhwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:08:29.191142 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:29.191054 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" podUID="da26a005-8c1e-46a4-8a17-93119a8a735d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 03 20:08:30.190058 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:30.190024 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zjhwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:08:30.190570 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:30.190101 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" podUID="da26a005-8c1e-46a4-8a17-93119a8a735d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 03 20:08:31.189485 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:31.189453 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zjhwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:08:31.189663 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:31.189507 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" podUID="da26a005-8c1e-46a4-8a17-93119a8a735d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 03 20:08:32.190117 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:32.190077 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zjhwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:08:32.190573 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:32.190142 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" podUID="da26a005-8c1e-46a4-8a17-93119a8a735d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 03 20:08:33.190200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:33.190140 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zjhwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:08:33.190618 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:33.190239 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" podUID="da26a005-8c1e-46a4-8a17-93119a8a735d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 03 20:08:34.190250 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:34.190216 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zjhwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:08:34.190644 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:34.190280 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" podUID="da26a005-8c1e-46a4-8a17-93119a8a735d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 03 20:08:35.189599 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:35.189563 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zjhwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:08:35.189803 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:35.189623 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" podUID="da26a005-8c1e-46a4-8a17-93119a8a735d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 03 20:08:36.189546 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:36.189513 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zjhwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:08:36.189972 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:36.189565 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" podUID="da26a005-8c1e-46a4-8a17-93119a8a735d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 03 20:08:37.193828 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:37.193796 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:37.194238 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:37.194140 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:37.195004 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:37.194985 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:08:38.876469 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:38.876434 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c44549c46-lcdq6"] Nov 03 20:08:38.880226 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:38.880196 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c44549c46-lcdq6" Nov 03 20:08:38.881074 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:38.881050 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c44549c46-lcdq6"] Nov 03 20:08:38.885725 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:38.885696 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 20:08:38.885725 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:38.885706 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 20:08:38.885965 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:38.885923 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 20:08:38.886071 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:38.885981 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-grh7n\"" Nov 03 20:08:38.927691 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:38.927654 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7knk\" (UniqueName: \"kubernetes.io/projected/8b5289a1-1b79-4406-9b5b-a354caae1c97-kube-api-access-m7knk\") pod \"authorino-5c44549c46-lcdq6\" (UID: \"8b5289a1-1b79-4406-9b5b-a354caae1c97\") " pod="kuadrant-system/authorino-5c44549c46-lcdq6" Nov 03 20:08:38.927875 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:38.927805 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8b5289a1-1b79-4406-9b5b-a354caae1c97-tls-cert\") pod \"authorino-5c44549c46-lcdq6\" (UID: \"8b5289a1-1b79-4406-9b5b-a354caae1c97\") " pod="kuadrant-system/authorino-5c44549c46-lcdq6" Nov 03 20:08:39.029266 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:39.029233 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7knk\" (UniqueName: \"kubernetes.io/projected/8b5289a1-1b79-4406-9b5b-a354caae1c97-kube-api-access-m7knk\") pod \"authorino-5c44549c46-lcdq6\" (UID: \"8b5289a1-1b79-4406-9b5b-a354caae1c97\") " pod="kuadrant-system/authorino-5c44549c46-lcdq6" Nov 03 20:08:39.029439 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:39.029349 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8b5289a1-1b79-4406-9b5b-a354caae1c97-tls-cert\") pod \"authorino-5c44549c46-lcdq6\" (UID: \"8b5289a1-1b79-4406-9b5b-a354caae1c97\") " pod="kuadrant-system/authorino-5c44549c46-lcdq6" Nov 03 20:08:39.031766 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:39.031738 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8b5289a1-1b79-4406-9b5b-a354caae1c97-tls-cert\") pod \"authorino-5c44549c46-lcdq6\" (UID: \"8b5289a1-1b79-4406-9b5b-a354caae1c97\") " pod="kuadrant-system/authorino-5c44549c46-lcdq6" Nov 03 20:08:39.041899 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:39.041877 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7knk\" (UniqueName: \"kubernetes.io/projected/8b5289a1-1b79-4406-9b5b-a354caae1c97-kube-api-access-m7knk\") pod \"authorino-5c44549c46-lcdq6\" (UID: \"8b5289a1-1b79-4406-9b5b-a354caae1c97\") " pod="kuadrant-system/authorino-5c44549c46-lcdq6" Nov 03 20:08:39.192434 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:39.192346 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c44549c46-lcdq6" Nov 03 20:08:39.471350 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:39.471326 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c44549c46-lcdq6"] Nov 03 20:08:39.474035 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:08:39.474000 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b5289a1_1b79_4406_9b5b_a354caae1c97.slice/crio-d99123d9d36dc2d3d5fecce0a76edcff20ead2caef4624f38b6de5e03aa58d38 WatchSource:0}: Error finding container d99123d9d36dc2d3d5fecce0a76edcff20ead2caef4624f38b6de5e03aa58d38: Status 404 returned error can't find the container with id d99123d9d36dc2d3d5fecce0a76edcff20ead2caef4624f38b6de5e03aa58d38 Nov 03 20:08:39.923530 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:39.923496 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq"] Nov 03 20:08:39.928682 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:39.928655 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:39.942319 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:39.942291 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq"] Nov 03 20:08:40.037541 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.037497 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea43d6b7-986b-4194-b9b3-b86e22e86544-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.037541 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.037551 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea43d6b7-986b-4194-b9b3-b86e22e86544-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.037768 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.037613 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea43d6b7-986b-4194-b9b3-b86e22e86544-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.037768 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.037634 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea43d6b7-986b-4194-b9b3-b86e22e86544-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.037768 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.037717 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea43d6b7-986b-4194-b9b3-b86e22e86544-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.037866 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.037770 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea43d6b7-986b-4194-b9b3-b86e22e86544-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.037866 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.037841 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea43d6b7-986b-4194-b9b3-b86e22e86544-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.037930 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.037871 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea43d6b7-986b-4194-b9b3-b86e22e86544-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.037930 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.037892 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntvdd\" (UniqueName: \"kubernetes.io/projected/ea43d6b7-986b-4194-b9b3-b86e22e86544-kube-api-access-ntvdd\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.127428 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.127389 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c44549c46-lcdq6" event={"ID":"8b5289a1-1b79-4406-9b5b-a354caae1c97","Type":"ContainerStarted","Data":"d99123d9d36dc2d3d5fecce0a76edcff20ead2caef4624f38b6de5e03aa58d38"} Nov 03 20:08:40.139205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.139148 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea43d6b7-986b-4194-b9b3-b86e22e86544-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.139379 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.139239 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea43d6b7-986b-4194-b9b3-b86e22e86544-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.139379 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.139271 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntvdd\" (UniqueName: \"kubernetes.io/projected/ea43d6b7-986b-4194-b9b3-b86e22e86544-kube-api-access-ntvdd\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.139521 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.139405 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea43d6b7-986b-4194-b9b3-b86e22e86544-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.139521 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.139447 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea43d6b7-986b-4194-b9b3-b86e22e86544-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.139521 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.139486 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea43d6b7-986b-4194-b9b3-b86e22e86544-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.139521 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.139510 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea43d6b7-986b-4194-b9b3-b86e22e86544-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.139738 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.139532 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea43d6b7-986b-4194-b9b3-b86e22e86544-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.139738 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.139596 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea43d6b7-986b-4194-b9b3-b86e22e86544-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.139871 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.139850 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea43d6b7-986b-4194-b9b3-b86e22e86544-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.139961 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.139933 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea43d6b7-986b-4194-b9b3-b86e22e86544-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.140040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.140013 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea43d6b7-986b-4194-b9b3-b86e22e86544-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.140106 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.140043 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea43d6b7-986b-4194-b9b3-b86e22e86544-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.140511 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.140489 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea43d6b7-986b-4194-b9b3-b86e22e86544-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.142048 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.142023 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea43d6b7-986b-4194-b9b3-b86e22e86544-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.142221 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.142199 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea43d6b7-986b-4194-b9b3-b86e22e86544-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.151930 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.151897 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea43d6b7-986b-4194-b9b3-b86e22e86544-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.152045 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.151971 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntvdd\" (UniqueName: \"kubernetes.io/projected/ea43d6b7-986b-4194-b9b3-b86e22e86544-kube-api-access-ntvdd\") pod \"maas-default-gateway-openshift-default-77f556d647-t7zgq\" (UID: \"ea43d6b7-986b-4194-b9b3-b86e22e86544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.247307 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.247221 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:40.532285 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.532261 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq"] Nov 03 20:08:40.534247 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:08:40.534215 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea43d6b7_986b_4194_b9b3_b86e22e86544.slice/crio-58dbc472955d4a39c6c7df69febcc4a9b40b194ea1504d52fc6558e7465d0a82 WatchSource:0}: Error finding container 58dbc472955d4a39c6c7df69febcc4a9b40b194ea1504d52fc6558e7465d0a82: Status 404 returned error can't find the container with id 58dbc472955d4a39c6c7df69febcc4a9b40b194ea1504d52fc6558e7465d0a82 Nov 03 20:08:40.537331 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.537298 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:08:40.537432 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.537373 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:08:40.537432 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:40.537418 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:08:41.164678 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:41.164559 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c44549c46-lcdq6" event={"ID":"8b5289a1-1b79-4406-9b5b-a354caae1c97","Type":"ContainerStarted","Data":"3fb6595ae43fb2bd83aa7e10711db3b865e7ba6186df560342a520570dd6f9b1"} Nov 03 20:08:41.167846 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:41.167815 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" event={"ID":"ea43d6b7-986b-4194-b9b3-b86e22e86544","Type":"ContainerStarted","Data":"488b50ac15ab295ddb1da02a82c6288936de7df72913a3ae37f590422547539c"} Nov 03 20:08:41.168007 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:41.167871 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" event={"ID":"ea43d6b7-986b-4194-b9b3-b86e22e86544","Type":"ContainerStarted","Data":"58dbc472955d4a39c6c7df69febcc4a9b40b194ea1504d52fc6558e7465d0a82"} Nov 03 20:08:41.184089 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:41.184023 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5c44549c46-lcdq6" podStartSLOduration=1.833171906 podStartE2EDuration="3.18400702s" totalImagesPullingTime="1.350835114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:41.183243474 +0000 UTC m=+704.115110986" watchObservedRunningTime="2026-11-03 20:08:41.18400702 +0000 UTC m=+704.115874532" Nov 03 20:08:41.205624 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:41.205564 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" podStartSLOduration=2.205544639 podStartE2EDuration="2.205544639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:41.204717746 +0000 UTC m=+704.136585259" watchObservedRunningTime="2026-11-03 20:08:41.205544639 +0000 UTC m=+704.137412152" Nov 03 20:08:41.270633 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:41.269210 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:41.297339 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:41.297299 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-t7zgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 20:08:41.297632 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:41.297598 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" podUID="ea43d6b7-986b-4194-b9b3-b86e22e86544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 20:08:42.256036 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:42.255568 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-t7zgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 20:08:42.256036 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:42.255646 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" podUID="ea43d6b7-986b-4194-b9b3-b86e22e86544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 20:08:43.248387 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:43.248343 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-t7zgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 20:08:43.248593 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:43.248403 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" podUID="ea43d6b7-986b-4194-b9b3-b86e22e86544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 20:08:44.247868 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:44.247827 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-t7zgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 20:08:44.248370 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:44.247891 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" podUID="ea43d6b7-986b-4194-b9b3-b86e22e86544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 20:08:45.248542 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:45.248506 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-t7zgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 20:08:45.248933 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:45.248591 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" podUID="ea43d6b7-986b-4194-b9b3-b86e22e86544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 20:08:46.248265 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:46.248228 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-t7zgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 20:08:46.248548 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:46.248519 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" podUID="ea43d6b7-986b-4194-b9b3-b86e22e86544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 20:08:47.248053 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:47.248007 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-t7zgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 20:08:47.248651 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:47.248614 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" podUID="ea43d6b7-986b-4194-b9b3-b86e22e86544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 20:08:47.391410 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:47.391368 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-h8g6d"] Nov 03 20:08:47.395109 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:47.395087 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-h8g6d" Nov 03 20:08:47.412993 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:47.412963 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-h8g6d"] Nov 03 20:08:47.456647 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:47.456591 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb824440-b2b0-478f-a06d-acd897fceb6d-plugins-config-volume\") pod \"payload-processing-7bb9489676-h8g6d\" (UID: \"cb824440-b2b0-478f-a06d-acd897fceb6d\") " pod="openshift-ingress/payload-processing-7bb9489676-h8g6d" Nov 03 20:08:47.456993 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:47.456962 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hjgr\" (UniqueName: \"kubernetes.io/projected/cb824440-b2b0-478f-a06d-acd897fceb6d-kube-api-access-8hjgr\") pod \"payload-processing-7bb9489676-h8g6d\" (UID: \"cb824440-b2b0-478f-a06d-acd897fceb6d\") " pod="openshift-ingress/payload-processing-7bb9489676-h8g6d" Nov 03 20:08:47.470112 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:47.470088 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 20:08:47.470275 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:47.470103 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-wgb65\"" Nov 03 20:08:47.557572 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:47.557465 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hjgr\" (UniqueName: \"kubernetes.io/projected/cb824440-b2b0-478f-a06d-acd897fceb6d-kube-api-access-8hjgr\") pod \"payload-processing-7bb9489676-h8g6d\" (UID: \"cb824440-b2b0-478f-a06d-acd897fceb6d\") " pod="openshift-ingress/payload-processing-7bb9489676-h8g6d" Nov 03 20:08:47.557572 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:47.557545 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb824440-b2b0-478f-a06d-acd897fceb6d-plugins-config-volume\") pod \"payload-processing-7bb9489676-h8g6d\" (UID: \"cb824440-b2b0-478f-a06d-acd897fceb6d\") " pod="openshift-ingress/payload-processing-7bb9489676-h8g6d" Nov 03 20:08:47.558302 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:47.558276 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb824440-b2b0-478f-a06d-acd897fceb6d-plugins-config-volume\") pod \"payload-processing-7bb9489676-h8g6d\" (UID: \"cb824440-b2b0-478f-a06d-acd897fceb6d\") " pod="openshift-ingress/payload-processing-7bb9489676-h8g6d" Nov 03 20:08:47.570224 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:47.570192 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hjgr\" (UniqueName: \"kubernetes.io/projected/cb824440-b2b0-478f-a06d-acd897fceb6d-kube-api-access-8hjgr\") pod \"payload-processing-7bb9489676-h8g6d\" (UID: \"cb824440-b2b0-478f-a06d-acd897fceb6d\") " pod="openshift-ingress/payload-processing-7bb9489676-h8g6d" Nov 03 20:08:47.709532 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:47.709489 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-h8g6d" Nov 03 20:08:48.033200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:48.030080 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-h8g6d"] Nov 03 20:08:48.237885 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:48.237244 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-h8g6d" event={"ID":"cb824440-b2b0-478f-a06d-acd897fceb6d","Type":"ContainerStarted","Data":"0ea65266bcc6e3f15a62d7914ff4d1837bc543c7dce2ebbf0bda798d23c747a9"} Nov 03 20:08:49.248764 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:49.248728 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-t7zgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:08:49.249217 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:49.248800 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" podUID="ea43d6b7-986b-4194-b9b3-b86e22e86544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:50.249352 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:50.249304 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-t7zgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:08:50.249770 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:50.249385 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" podUID="ea43d6b7-986b-4194-b9b3-b86e22e86544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:51.266888 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:51.266839 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-h8g6d" event={"ID":"cb824440-b2b0-478f-a06d-acd897fceb6d","Type":"ContainerStarted","Data":"925a331276126031f8828d7a8900c8b808979fecd883df2a08cdddc6aa1fb330"} Nov 03 20:08:51.267449 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:51.266999 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-h8g6d" Nov 03 20:08:51.287669 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:51.287618 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-h8g6d" podStartSLOduration=1.999897685 podStartE2EDuration="4.287603945s" totalImagesPullingTime="2.28770626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:51.285715197 +0000 UTC m=+714.217582719" watchObservedRunningTime="2026-11-03 20:08:51.287603945 +0000 UTC m=+714.219471473" Nov 03 20:08:51.290311 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:51.290290 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-t7zgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:08:51.290404 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:51.290330 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" podUID="ea43d6b7-986b-4194-b9b3-b86e22e86544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:51.505498 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:51.505434 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:51.506368 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:51.506332 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:51.507083 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:51.507054 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" Nov 03 20:08:54.927517 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:54.927479 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4"] Nov 03 20:08:54.932230 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:54.932173 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:54.944687 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:54.944659 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4"] Nov 03 20:08:55.033053 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.033021 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea819fe0-461b-4952-a40e-7a47e8f51580-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.033053 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.033062 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea819fe0-461b-4952-a40e-7a47e8f51580-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.033313 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.033133 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea819fe0-461b-4952-a40e-7a47e8f51580-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.033313 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.033220 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea819fe0-461b-4952-a40e-7a47e8f51580-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.033313 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.033268 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47hbs\" (UniqueName: \"kubernetes.io/projected/ea819fe0-461b-4952-a40e-7a47e8f51580-kube-api-access-47hbs\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.033462 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.033370 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea819fe0-461b-4952-a40e-7a47e8f51580-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.033462 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.033424 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea819fe0-461b-4952-a40e-7a47e8f51580-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.033544 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.033476 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea819fe0-461b-4952-a40e-7a47e8f51580-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.033544 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.033505 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea819fe0-461b-4952-a40e-7a47e8f51580-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.134346 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.134306 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea819fe0-461b-4952-a40e-7a47e8f51580-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.134543 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.134371 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea819fe0-461b-4952-a40e-7a47e8f51580-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.134543 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.134393 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea819fe0-461b-4952-a40e-7a47e8f51580-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.134543 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.134422 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea819fe0-461b-4952-a40e-7a47e8f51580-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.134543 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.134445 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea819fe0-461b-4952-a40e-7a47e8f51580-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.134543 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.134462 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea819fe0-461b-4952-a40e-7a47e8f51580-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.134543 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.134487 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea819fe0-461b-4952-a40e-7a47e8f51580-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.134543 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.134523 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea819fe0-461b-4952-a40e-7a47e8f51580-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.134905 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.134579 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47hbs\" (UniqueName: \"kubernetes.io/projected/ea819fe0-461b-4952-a40e-7a47e8f51580-kube-api-access-47hbs\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.134905 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.134877 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea819fe0-461b-4952-a40e-7a47e8f51580-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.135014 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.134992 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea819fe0-461b-4952-a40e-7a47e8f51580-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.135171 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.135151 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea819fe0-461b-4952-a40e-7a47e8f51580-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.135261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.135216 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea819fe0-461b-4952-a40e-7a47e8f51580-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.135499 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.135477 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea819fe0-461b-4952-a40e-7a47e8f51580-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.137147 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.137127 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea819fe0-461b-4952-a40e-7a47e8f51580-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.137290 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.137272 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea819fe0-461b-4952-a40e-7a47e8f51580-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.147922 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.147894 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea819fe0-461b-4952-a40e-7a47e8f51580-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.148209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.148169 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47hbs\" (UniqueName: \"kubernetes.io/projected/ea819fe0-461b-4952-a40e-7a47e8f51580-kube-api-access-47hbs\") pod \"maas-default-gateway-openshift-default-77f556d647-8bvn4\" (UID: \"ea819fe0-461b-4952-a40e-7a47e8f51580\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.245575 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.245485 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:55.529821 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.529791 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4"] Nov 03 20:08:55.531668 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:08:55.531642 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea819fe0_461b_4952_a40e_7a47e8f51580.slice/crio-7177e8e9859b0e8c95f7b92b563ea6058804a9b23be1aa4c78f92fa9c55ae697 WatchSource:0}: Error finding container 7177e8e9859b0e8c95f7b92b563ea6058804a9b23be1aa4c78f92fa9c55ae697: Status 404 returned error can't find the container with id 7177e8e9859b0e8c95f7b92b563ea6058804a9b23be1aa4c78f92fa9c55ae697 Nov 03 20:08:55.534604 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.534570 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:08:55.534676 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.534636 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:08:55.534676 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:55.534671 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:08:56.292929 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:56.292885 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" event={"ID":"ea819fe0-461b-4952-a40e-7a47e8f51580","Type":"ContainerStarted","Data":"f1c53946e1d11afbfae748f2d119b40c7ee9aa2370defc25a326328ccc7ac382"} Nov 03 20:08:56.293319 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:56.292947 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" event={"ID":"ea819fe0-461b-4952-a40e-7a47e8f51580","Type":"ContainerStarted","Data":"7177e8e9859b0e8c95f7b92b563ea6058804a9b23be1aa4c78f92fa9c55ae697"} Nov 03 20:08:56.334202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:56.318631 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" podStartSLOduration=2.318614017 podStartE2EDuration="2.318614017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:56.316703514 +0000 UTC m=+719.248571019" watchObservedRunningTime="2026-11-03 20:08:56.318614017 +0000 UTC m=+719.250481528" Nov 03 20:08:57.246163 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:57.246126 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:08:57.247621 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:57.247593 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8bvn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:08:57.247735 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:57.247641 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" podUID="ea819fe0-461b-4952-a40e-7a47e8f51580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 20:08:58.207662 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:08:58.207612 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:08:58.245881 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:58.245846 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8bvn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:08:58.246070 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:58.245900 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" podUID="ea819fe0-461b-4952-a40e-7a47e8f51580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 20:08:59.246654 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:59.246620 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8bvn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:08:59.247088 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:08:59.246672 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" podUID="ea819fe0-461b-4952-a40e-7a47e8f51580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 20:09:00.246477 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:09:00.246439 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8bvn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:09:00.246662 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:09:00.246497 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" podUID="ea819fe0-461b-4952-a40e-7a47e8f51580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 20:09:01.246057 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:09:01.246020 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8bvn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:09:01.246562 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:09:01.246074 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" podUID="ea819fe0-461b-4952-a40e-7a47e8f51580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 20:09:02.246137 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:09:02.246094 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8bvn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:09:02.246542 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:09:02.246207 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" podUID="ea819fe0-461b-4952-a40e-7a47e8f51580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 20:09:02.275727 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:09:02.275701 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-h8g6d" Nov 03 20:09:03.883170 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:09:03.883133 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:09:03.883936 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:09:03.883910 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:09:03.884542 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:09:03.884525 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" Nov 03 20:09:28.207294 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:09:28.207255 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:09:58.207362 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:09:58.207321 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:10:00.491574 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.491535 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr"] Nov 03 20:10:00.495231 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.495210 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.500606 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.500588 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3aba31-openshift-default-dockercfg-s4h77\"" Nov 03 20:10:00.508362 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.508336 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr"] Nov 03 20:10:00.574812 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.574768 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.574991 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.574823 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.575063 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.574979 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.575063 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.575041 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7vpd\" (UniqueName: \"kubernetes.io/projected/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-kube-api-access-t7vpd\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.575137 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.575111 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.575206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.575140 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.575206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.575165 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.575323 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.575274 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.575378 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.575351 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.675913 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.675879 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.676090 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.675939 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7vpd\" (UniqueName: \"kubernetes.io/projected/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-kube-api-access-t7vpd\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.676090 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.675971 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.676090 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.675997 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.676090 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.676032 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.676090 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.676057 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.676381 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.676098 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.676381 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.676140 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.676381 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.676205 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.676588 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.676563 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.676653 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.676628 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.676707 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.676676 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.676749 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.676739 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.676842 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.676821 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.678227 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.678203 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.678540 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.678521 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.689956 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.689934 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7vpd\" (UniqueName: \"kubernetes.io/projected/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-kube-api-access-t7vpd\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.690672 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.690648 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.808832 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.808746 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:00.851344 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.851303 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j"] Nov 03 20:10:00.855998 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.855966 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:00.869063 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.869032 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j"] Nov 03 20:10:00.901588 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.901560 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-a6f98b-openshift-default-dockercfg-xrwzw\"" Nov 03 20:10:00.979078 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.979032 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:00.979078 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.979079 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:00.979292 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.979191 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:00.979292 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.979228 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:00.979292 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.979276 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:00.979390 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.979333 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc569\" (UniqueName: \"kubernetes.io/projected/b92265eb-94c4-42c6-b2c0-b22feab2eb07-kube-api-access-rc569\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:00.979390 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.979368 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:00.979481 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.979459 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:00.979565 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:00.979547 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:01.080620 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.080583 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:01.080817 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.080633 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:01.080817 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.080653 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:01.080817 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.080694 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:01.080817 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.080716 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:01.080817 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.080749 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:01.080817 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.080807 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc569\" (UniqueName: \"kubernetes.io/projected/b92265eb-94c4-42c6-b2c0-b22feab2eb07-kube-api-access-rc569\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:01.081133 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.080838 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:01.081133 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.080883 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:01.081133 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.081027 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:01.081435 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.081406 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:01.081623 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.081428 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:01.081714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.081450 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:01.081844 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.081821 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:01.083097 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.083077 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:01.083479 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.083462 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:01.095251 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.095229 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:01.095337 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.095290 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc569\" (UniqueName: \"kubernetes.io/projected/b92265eb-94c4-42c6-b2c0-b22feab2eb07-kube-api-access-rc569\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:01.101170 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.101141 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr"] Nov 03 20:10:01.103208 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:10:01.103172 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645fc172_d7a8_48a6_93ec_6c9ed0fc0304.slice/crio-dee726375bc591efcedb045721d4ce49e39bce600d9ce06af4555f13cd4514fc WatchSource:0}: Error finding container dee726375bc591efcedb045721d4ce49e39bce600d9ce06af4555f13cd4514fc: Status 404 returned error can't find the container with id dee726375bc591efcedb045721d4ce49e39bce600d9ce06af4555f13cd4514fc Nov 03 20:10:01.105455 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.105427 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:10:01.105514 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.105487 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:10:01.105554 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.105517 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:10:01.176315 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.176291 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:01.465295 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.465265 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j"] Nov 03 20:10:01.467115 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:10:01.467090 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92265eb_94c4_42c6_b2c0_b22feab2eb07.slice/crio-38f117737cfe7b1c75d7c91f4c8d9a6c18312a338d35ca21ebb7d27251842972 WatchSource:0}: Error finding container 38f117737cfe7b1c75d7c91f4c8d9a6c18312a338d35ca21ebb7d27251842972: Status 404 returned error can't find the container with id 38f117737cfe7b1c75d7c91f4c8d9a6c18312a338d35ca21ebb7d27251842972 Nov 03 20:10:01.469408 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.469375 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:10:01.469496 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.469452 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:10:01.469552 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.469495 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:10:01.598766 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.598717 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" event={"ID":"645fc172-d7a8-48a6-93ec-6c9ed0fc0304","Type":"ContainerStarted","Data":"07c212a00a781a6e576aa7189d906443fea4dbd04b72779b4a403a8180aecb35"} Nov 03 20:10:01.599245 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.598799 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" event={"ID":"645fc172-d7a8-48a6-93ec-6c9ed0fc0304","Type":"ContainerStarted","Data":"dee726375bc591efcedb045721d4ce49e39bce600d9ce06af4555f13cd4514fc"} Nov 03 20:10:01.600603 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.600572 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" event={"ID":"b92265eb-94c4-42c6-b2c0-b22feab2eb07","Type":"ContainerStarted","Data":"91477df13a4f8468eabf40a759fe67b4c0a795667559c47f983c329aec919025"} Nov 03 20:10:01.600725 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.600617 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" event={"ID":"b92265eb-94c4-42c6-b2c0-b22feab2eb07","Type":"ContainerStarted","Data":"38f117737cfe7b1c75d7c91f4c8d9a6c18312a338d35ca21ebb7d27251842972"} Nov 03 20:10:01.624295 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.624229 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" podStartSLOduration=1.624209003 podStartE2EDuration="1.624209003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:01.621734017 +0000 UTC m=+784.553601526" watchObservedRunningTime="2026-11-03 20:10:01.624209003 +0000 UTC m=+784.556076515" Nov 03 20:10:01.646661 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.646590 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" podStartSLOduration=1.64657069 podStartE2EDuration="1.64657069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:01.646250378 +0000 UTC m=+784.578117890" watchObservedRunningTime="2026-11-03 20:10:01.64657069 +0000 UTC m=+784.578438202" Nov 03 20:10:01.809399 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:01.809315 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:02.177260 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:02.177223 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:02.182805 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:02.182771 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:02.605720 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:02.605681 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:02.606556 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:02.606525 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:02.814082 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:02.814052 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:03.610249 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:03.610215 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:03.611166 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:03.611147 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:05.294312 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:05.294271 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp"] Nov 03 20:10:05.298295 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:05.298271 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp" Nov 03 20:10:05.306902 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:05.306871 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp"] Nov 03 20:10:05.308387 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:05.308363 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-a6f98b\"" Nov 03 20:10:05.308508 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:05.308394 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-a6f98b-dockercfg-nqgf7\"" Nov 03 20:10:05.423700 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:05.423661 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ae2662-2a94-411f-a9e1-a0fb1452f3ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp\" (UID: \"e2ae2662-2a94-411f-a9e1-a0fb1452f3ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp" Nov 03 20:10:05.423894 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:05.423783 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4d92\" (UniqueName: \"kubernetes.io/projected/e2ae2662-2a94-411f-a9e1-a0fb1452f3ae-kube-api-access-z4d92\") pod \"payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp\" (UID: \"e2ae2662-2a94-411f-a9e1-a0fb1452f3ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp" Nov 03 20:10:05.524730 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:05.524688 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ae2662-2a94-411f-a9e1-a0fb1452f3ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp\" (UID: \"e2ae2662-2a94-411f-a9e1-a0fb1452f3ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp" Nov 03 20:10:05.524883 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:05.524786 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4d92\" (UniqueName: \"kubernetes.io/projected/e2ae2662-2a94-411f-a9e1-a0fb1452f3ae-kube-api-access-z4d92\") pod \"payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp\" (UID: \"e2ae2662-2a94-411f-a9e1-a0fb1452f3ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp" Nov 03 20:10:05.525495 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:05.525472 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ae2662-2a94-411f-a9e1-a0fb1452f3ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp\" (UID: \"e2ae2662-2a94-411f-a9e1-a0fb1452f3ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp" Nov 03 20:10:05.538575 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:05.538550 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4d92\" (UniqueName: \"kubernetes.io/projected/e2ae2662-2a94-411f-a9e1-a0fb1452f3ae-kube-api-access-z4d92\") pod \"payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp\" (UID: \"e2ae2662-2a94-411f-a9e1-a0fb1452f3ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp" Nov 03 20:10:05.613692 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:05.613665 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp" Nov 03 20:10:05.890550 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:05.890521 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp"] Nov 03 20:10:05.893305 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:10:05.893278 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ae2662_2a94_411f_a9e1_a0fb1452f3ae.slice/crio-bc40aa39652a4129ec16ffb2bd3b9d5b11489b354b2faa2ad47a1343723b8140 WatchSource:0}: Error finding container bc40aa39652a4129ec16ffb2bd3b9d5b11489b354b2faa2ad47a1343723b8140: Status 404 returned error can't find the container with id bc40aa39652a4129ec16ffb2bd3b9d5b11489b354b2faa2ad47a1343723b8140 Nov 03 20:10:05.935776 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:05.935748 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt"] Nov 03 20:10:05.940978 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:05.940460 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" Nov 03 20:10:05.954123 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:05.954100 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-a6f98b-serving-cert\"" Nov 03 20:10:05.970595 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:05.970571 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt"] Nov 03 20:10:06.031224 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.031166 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27lvt\" (UniqueName: \"kubernetes.io/projected/d7646487-52b3-4fd9-8781-2d440af503a2-kube-api-access-27lvt\") pod \"maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt\" (UID: \"d7646487-52b3-4fd9-8781-2d440af503a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" Nov 03 20:10:06.031404 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.031342 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7646487-52b3-4fd9-8781-2d440af503a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt\" (UID: \"d7646487-52b3-4fd9-8781-2d440af503a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" Nov 03 20:10:06.031466 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.031408 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7646487-52b3-4fd9-8781-2d440af503a2-maas-api-tls\") pod \"maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt\" (UID: \"d7646487-52b3-4fd9-8781-2d440af503a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" Nov 03 20:10:06.031506 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.031493 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7646487-52b3-4fd9-8781-2d440af503a2-metrics-certs\") pod \"maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt\" (UID: \"d7646487-52b3-4fd9-8781-2d440af503a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" Nov 03 20:10:06.133153 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.133113 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7646487-52b3-4fd9-8781-2d440af503a2-maas-api-tls\") pod \"maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt\" (UID: \"d7646487-52b3-4fd9-8781-2d440af503a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" Nov 03 20:10:06.133395 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.133172 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7646487-52b3-4fd9-8781-2d440af503a2-metrics-certs\") pod \"maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt\" (UID: \"d7646487-52b3-4fd9-8781-2d440af503a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" Nov 03 20:10:06.133395 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.133316 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27lvt\" (UniqueName: \"kubernetes.io/projected/d7646487-52b3-4fd9-8781-2d440af503a2-kube-api-access-27lvt\") pod \"maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt\" (UID: \"d7646487-52b3-4fd9-8781-2d440af503a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" Nov 03 20:10:06.133395 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:10:06.133357 4918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-serving-cert: secret "maas-api-e2e-worker-w5-a6f98b-serving-cert" not found Nov 03 20:10:06.133572 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.133399 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7646487-52b3-4fd9-8781-2d440af503a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt\" (UID: \"d7646487-52b3-4fd9-8781-2d440af503a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" Nov 03 20:10:06.133572 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:10:06.133438 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7646487-52b3-4fd9-8781-2d440af503a2-maas-api-tls podName:d7646487-52b3-4fd9-8781-2d440af503a2 nodeName:}" failed. No retries permitted until 2026-11-03 20:10:06.633413418 +0000 UTC m=+789.565280925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d7646487-52b3-4fd9-8781-2d440af503a2-maas-api-tls") pod "maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" (UID: "d7646487-52b3-4fd9-8781-2d440af503a2") : secret "maas-api-e2e-worker-w5-a6f98b-serving-cert" not found Nov 03 20:10:06.134239 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.134211 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7646487-52b3-4fd9-8781-2d440af503a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt\" (UID: \"d7646487-52b3-4fd9-8781-2d440af503a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" Nov 03 20:10:06.135927 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.135902 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7646487-52b3-4fd9-8781-2d440af503a2-metrics-certs\") pod \"maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt\" (UID: \"d7646487-52b3-4fd9-8781-2d440af503a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" Nov 03 20:10:06.145901 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.145853 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27lvt\" (UniqueName: \"kubernetes.io/projected/d7646487-52b3-4fd9-8781-2d440af503a2-kube-api-access-27lvt\") pod \"maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt\" (UID: \"d7646487-52b3-4fd9-8781-2d440af503a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" Nov 03 20:10:06.626440 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.626405 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp" event={"ID":"e2ae2662-2a94-411f-a9e1-a0fb1452f3ae","Type":"ContainerStarted","Data":"4fbab76bf14faf13fdaf355e672491d7013ac602bbeb3c18848fbbc8c21a41fa"} Nov 03 20:10:06.626847 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.626494 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp" event={"ID":"e2ae2662-2a94-411f-a9e1-a0fb1452f3ae","Type":"ContainerStarted","Data":"bc40aa39652a4129ec16ffb2bd3b9d5b11489b354b2faa2ad47a1343723b8140"} Nov 03 20:10:06.626847 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.626525 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp" Nov 03 20:10:06.639173 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.639141 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7646487-52b3-4fd9-8781-2d440af503a2-maas-api-tls\") pod \"maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt\" (UID: \"d7646487-52b3-4fd9-8781-2d440af503a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" Nov 03 20:10:06.641818 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.641780 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7646487-52b3-4fd9-8781-2d440af503a2-maas-api-tls\") pod \"maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt\" (UID: \"d7646487-52b3-4fd9-8781-2d440af503a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" Nov 03 20:10:06.655402 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.655350 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp" podStartSLOduration=1.655336959 podStartE2EDuration="1.655336959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:06.65303966 +0000 UTC m=+789.584907188" watchObservedRunningTime="2026-11-03 20:10:06.655336959 +0000 UTC m=+789.587204470" Nov 03 20:10:06.771456 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.771419 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x"] Nov 03 20:10:06.776440 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.776414 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x" Nov 03 20:10:06.787427 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.787402 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x"] Nov 03 20:10:06.804759 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.804727 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-b1b190-dockercfg-c7jm7\"" Nov 03 20:10:06.855876 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.855842 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-b1b190\"" Nov 03 20:10:06.879522 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.879420 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" Nov 03 20:10:06.943237 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.943205 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c47f933-d238-4576-b42e-36deda6d8528-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x\" (UID: \"0c47f933-d238-4576-b42e-36deda6d8528\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x" Nov 03 20:10:06.943652 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:06.943333 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52hwd\" (UniqueName: \"kubernetes.io/projected/0c47f933-d238-4576-b42e-36deda6d8528-kube-api-access-52hwd\") pod \"payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x\" (UID: \"0c47f933-d238-4576-b42e-36deda6d8528\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x" Nov 03 20:10:07.044207 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:07.044146 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c47f933-d238-4576-b42e-36deda6d8528-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x\" (UID: \"0c47f933-d238-4576-b42e-36deda6d8528\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x" Nov 03 20:10:07.044383 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:07.044238 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52hwd\" (UniqueName: \"kubernetes.io/projected/0c47f933-d238-4576-b42e-36deda6d8528-kube-api-access-52hwd\") pod \"payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x\" (UID: \"0c47f933-d238-4576-b42e-36deda6d8528\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x" Nov 03 20:10:07.044850 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:07.044828 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c47f933-d238-4576-b42e-36deda6d8528-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x\" (UID: \"0c47f933-d238-4576-b42e-36deda6d8528\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x" Nov 03 20:10:07.056668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:07.056639 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52hwd\" (UniqueName: \"kubernetes.io/projected/0c47f933-d238-4576-b42e-36deda6d8528-kube-api-access-52hwd\") pod \"payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x\" (UID: \"0c47f933-d238-4576-b42e-36deda6d8528\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x" Nov 03 20:10:07.093736 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:07.093701 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x" Nov 03 20:10:07.173486 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:07.173140 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt"] Nov 03 20:10:07.174864 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:10:07.174838 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7646487_52b3_4fd9_8781_2d440af503a2.slice/crio-ccf0e34d0b8467878428196995e62cac4b5dcf6637a00d2f4d8648c2587ae065 WatchSource:0}: Error finding container ccf0e34d0b8467878428196995e62cac4b5dcf6637a00d2f4d8648c2587ae065: Status 404 returned error can't find the container with id ccf0e34d0b8467878428196995e62cac4b5dcf6637a00d2f4d8648c2587ae065 Nov 03 20:10:07.378956 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:07.376239 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x"] Nov 03 20:10:07.379132 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:10:07.379108 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c47f933_d238_4576_b42e_36deda6d8528.slice/crio-77efe7a70e433bd8c748de89bc6e8f40ca332c8c9165f1b26349171df6e30e38 WatchSource:0}: Error finding container 77efe7a70e433bd8c748de89bc6e8f40ca332c8c9165f1b26349171df6e30e38: Status 404 returned error can't find the container with id 77efe7a70e433bd8c748de89bc6e8f40ca332c8c9165f1b26349171df6e30e38 Nov 03 20:10:07.633404 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:07.633372 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" event={"ID":"d7646487-52b3-4fd9-8781-2d440af503a2","Type":"ContainerStarted","Data":"ccf0e34d0b8467878428196995e62cac4b5dcf6637a00d2f4d8648c2587ae065"} Nov 03 20:10:07.634906 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:07.634881 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x" event={"ID":"0c47f933-d238-4576-b42e-36deda6d8528","Type":"ContainerStarted","Data":"a5111b615af2f7e9279ca27c21a8ab4bbf48e5f340a84b3b921940033d9ac16b"} Nov 03 20:10:07.635016 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:07.634928 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x" event={"ID":"0c47f933-d238-4576-b42e-36deda6d8528","Type":"ContainerStarted","Data":"77efe7a70e433bd8c748de89bc6e8f40ca332c8c9165f1b26349171df6e30e38"} Nov 03 20:10:07.635156 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:07.635140 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x" Nov 03 20:10:07.654511 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:07.654471 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x" podStartSLOduration=1.654458212 podStartE2EDuration="1.654458212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:07.652553795 +0000 UTC m=+790.584421305" watchObservedRunningTime="2026-11-03 20:10:07.654458212 +0000 UTC m=+790.586325723" Nov 03 20:10:08.213825 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.213786 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b"] Nov 03 20:10:08.218320 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.218297 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" Nov 03 20:10:08.226576 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.226344 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-90ef9c-serving-cert\"" Nov 03 20:10:08.230374 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.230347 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b"] Nov 03 20:10:08.358295 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.358236 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v58x\" (UniqueName: \"kubernetes.io/projected/4b47d59c-60aa-483d-82bd-35a5af6b10b6-kube-api-access-9v58x\") pod \"maas-api-e2e-worker-w2-90ef9c-5498778889-h885b\" (UID: \"4b47d59c-60aa-483d-82bd-35a5af6b10b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" Nov 03 20:10:08.358477 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.358317 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b47d59c-60aa-483d-82bd-35a5af6b10b6-metrics-certs\") pod \"maas-api-e2e-worker-w2-90ef9c-5498778889-h885b\" (UID: \"4b47d59c-60aa-483d-82bd-35a5af6b10b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" Nov 03 20:10:08.358477 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.358452 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b47d59c-60aa-483d-82bd-35a5af6b10b6-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-90ef9c-5498778889-h885b\" (UID: \"4b47d59c-60aa-483d-82bd-35a5af6b10b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" Nov 03 20:10:08.358640 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.358538 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b47d59c-60aa-483d-82bd-35a5af6b10b6-maas-api-tls\") pod \"maas-api-e2e-worker-w2-90ef9c-5498778889-h885b\" (UID: \"4b47d59c-60aa-483d-82bd-35a5af6b10b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" Nov 03 20:10:08.459748 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.459711 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v58x\" (UniqueName: \"kubernetes.io/projected/4b47d59c-60aa-483d-82bd-35a5af6b10b6-kube-api-access-9v58x\") pod \"maas-api-e2e-worker-w2-90ef9c-5498778889-h885b\" (UID: \"4b47d59c-60aa-483d-82bd-35a5af6b10b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" Nov 03 20:10:08.459748 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.459752 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b47d59c-60aa-483d-82bd-35a5af6b10b6-metrics-certs\") pod \"maas-api-e2e-worker-w2-90ef9c-5498778889-h885b\" (UID: \"4b47d59c-60aa-483d-82bd-35a5af6b10b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" Nov 03 20:10:08.459963 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.459781 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b47d59c-60aa-483d-82bd-35a5af6b10b6-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-90ef9c-5498778889-h885b\" (UID: \"4b47d59c-60aa-483d-82bd-35a5af6b10b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" Nov 03 20:10:08.459963 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.459814 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b47d59c-60aa-483d-82bd-35a5af6b10b6-maas-api-tls\") pod \"maas-api-e2e-worker-w2-90ef9c-5498778889-h885b\" (UID: \"4b47d59c-60aa-483d-82bd-35a5af6b10b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" Nov 03 20:10:08.460530 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.460506 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b47d59c-60aa-483d-82bd-35a5af6b10b6-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-90ef9c-5498778889-h885b\" (UID: \"4b47d59c-60aa-483d-82bd-35a5af6b10b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" Nov 03 20:10:08.463148 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.463125 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b47d59c-60aa-483d-82bd-35a5af6b10b6-maas-api-tls\") pod \"maas-api-e2e-worker-w2-90ef9c-5498778889-h885b\" (UID: \"4b47d59c-60aa-483d-82bd-35a5af6b10b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" Nov 03 20:10:08.463263 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.463207 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b47d59c-60aa-483d-82bd-35a5af6b10b6-metrics-certs\") pod \"maas-api-e2e-worker-w2-90ef9c-5498778889-h885b\" (UID: \"4b47d59c-60aa-483d-82bd-35a5af6b10b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" Nov 03 20:10:08.472305 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.472248 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v58x\" (UniqueName: \"kubernetes.io/projected/4b47d59c-60aa-483d-82bd-35a5af6b10b6-kube-api-access-9v58x\") pod \"maas-api-e2e-worker-w2-90ef9c-5498778889-h885b\" (UID: \"4b47d59c-60aa-483d-82bd-35a5af6b10b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" Nov 03 20:10:08.534236 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.534168 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" Nov 03 20:10:08.644124 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.644083 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" event={"ID":"d7646487-52b3-4fd9-8781-2d440af503a2","Type":"ContainerStarted","Data":"42f8d210a84f98baca99d18eb4c580ca90b3455b111588a53b31b2b91ae30b07"} Nov 03 20:10:08.644626 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.644339 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" Nov 03 20:10:08.666402 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.666341 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" podStartSLOduration=3.237906425 podStartE2EDuration="3.666323578s" totalImagesPullingTime="428.417153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:08.665724565 +0000 UTC m=+791.597592077" watchObservedRunningTime="2026-11-03 20:10:08.666323578 +0000 UTC m=+791.598191091" Nov 03 20:10:08.830543 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:08.830468 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b"] Nov 03 20:10:08.833052 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:10:08.833021 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b47d59c_60aa_483d_82bd_35a5af6b10b6.slice/crio-89c2eaf05d465cc55b355382086a8bb6b74531ffacfa5fbfa3e0d294b0fd6c3c WatchSource:0}: Error finding container 89c2eaf05d465cc55b355382086a8bb6b74531ffacfa5fbfa3e0d294b0fd6c3c: Status 404 returned error can't find the container with id 89c2eaf05d465cc55b355382086a8bb6b74531ffacfa5fbfa3e0d294b0fd6c3c Nov 03 20:10:09.650508 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:09.650469 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" event={"ID":"4b47d59c-60aa-483d-82bd-35a5af6b10b6","Type":"ContainerStarted","Data":"ad8a6f1fde75b55f94376445384ed67677a96db22d92248c86a4d8644364919e"} Nov 03 20:10:09.650940 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:09.650563 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" event={"ID":"4b47d59c-60aa-483d-82bd-35a5af6b10b6","Type":"ContainerStarted","Data":"89c2eaf05d465cc55b355382086a8bb6b74531ffacfa5fbfa3e0d294b0fd6c3c"} Nov 03 20:10:09.650940 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:09.650805 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" Nov 03 20:10:09.672796 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:09.672744 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" podStartSLOduration=1.241529249 podStartE2EDuration="1.672730753s" totalImagesPullingTime="431.201504ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:09.6711198 +0000 UTC m=+792.602987310" watchObservedRunningTime="2026-11-03 20:10:09.672730753 +0000 UTC m=+792.604598264" Nov 03 20:10:11.057404 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.057365 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx"] Nov 03 20:10:11.061284 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.061259 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" Nov 03 20:10:11.070897 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.070720 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-8013d9-serving-cert\"" Nov 03 20:10:11.075229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.075201 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx"] Nov 03 20:10:11.192694 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.192659 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74173946-29b1-4686-985c-73d7f43b51bf-maas-api-tls\") pod \"maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx\" (UID: \"74173946-29b1-4686-985c-73d7f43b51bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" Nov 03 20:10:11.192879 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.192715 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk6m9\" (UniqueName: \"kubernetes.io/projected/74173946-29b1-4686-985c-73d7f43b51bf-kube-api-access-dk6m9\") pod \"maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx\" (UID: \"74173946-29b1-4686-985c-73d7f43b51bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" Nov 03 20:10:11.192879 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.192828 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74173946-29b1-4686-985c-73d7f43b51bf-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx\" (UID: \"74173946-29b1-4686-985c-73d7f43b51bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" Nov 03 20:10:11.192949 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.192878 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74173946-29b1-4686-985c-73d7f43b51bf-metrics-certs\") pod \"maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx\" (UID: \"74173946-29b1-4686-985c-73d7f43b51bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" Nov 03 20:10:11.294024 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.293990 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74173946-29b1-4686-985c-73d7f43b51bf-maas-api-tls\") pod \"maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx\" (UID: \"74173946-29b1-4686-985c-73d7f43b51bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" Nov 03 20:10:11.294226 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.294041 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk6m9\" (UniqueName: \"kubernetes.io/projected/74173946-29b1-4686-985c-73d7f43b51bf-kube-api-access-dk6m9\") pod \"maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx\" (UID: \"74173946-29b1-4686-985c-73d7f43b51bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" Nov 03 20:10:11.294226 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.294098 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74173946-29b1-4686-985c-73d7f43b51bf-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx\" (UID: \"74173946-29b1-4686-985c-73d7f43b51bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" Nov 03 20:10:11.294226 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.294129 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74173946-29b1-4686-985c-73d7f43b51bf-metrics-certs\") pod \"maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx\" (UID: \"74173946-29b1-4686-985c-73d7f43b51bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" Nov 03 20:10:11.294799 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.294769 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74173946-29b1-4686-985c-73d7f43b51bf-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx\" (UID: \"74173946-29b1-4686-985c-73d7f43b51bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" Nov 03 20:10:11.296655 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.296627 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74173946-29b1-4686-985c-73d7f43b51bf-metrics-certs\") pod \"maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx\" (UID: \"74173946-29b1-4686-985c-73d7f43b51bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" Nov 03 20:10:11.296736 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.296627 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74173946-29b1-4686-985c-73d7f43b51bf-maas-api-tls\") pod \"maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx\" (UID: \"74173946-29b1-4686-985c-73d7f43b51bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" Nov 03 20:10:11.308024 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.307968 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk6m9\" (UniqueName: \"kubernetes.io/projected/74173946-29b1-4686-985c-73d7f43b51bf-kube-api-access-dk6m9\") pod \"maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx\" (UID: \"74173946-29b1-4686-985c-73d7f43b51bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" Nov 03 20:10:11.377968 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.377930 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" Nov 03 20:10:11.683427 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.683395 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx"] Nov 03 20:10:11.685657 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:10:11.685629 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74173946_29b1_4686_985c_73d7f43b51bf.slice/crio-dd710af59ed0b444412857793e4974154f355f63a442d69383d810bc1a17e306 WatchSource:0}: Error finding container dd710af59ed0b444412857793e4974154f355f63a442d69383d810bc1a17e306: Status 404 returned error can't find the container with id dd710af59ed0b444412857793e4974154f355f63a442d69383d810bc1a17e306 Nov 03 20:10:11.850079 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.850040 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b"] Nov 03 20:10:11.854097 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.854073 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" Nov 03 20:10:11.857557 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.857529 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-d412cf-serving-cert\"" Nov 03 20:10:11.865398 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:11.865354 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b"] Nov 03 20:10:12.001044 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:12.000968 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxftx\" (UniqueName: \"kubernetes.io/projected/a1c85e9e-0e22-4860-8859-b36713dcbcfc-kube-api-access-bxftx\") pod \"maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b\" (UID: \"a1c85e9e-0e22-4860-8859-b36713dcbcfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" Nov 03 20:10:12.001224 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:12.001152 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1c85e9e-0e22-4860-8859-b36713dcbcfc-metrics-certs\") pod \"maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b\" (UID: \"a1c85e9e-0e22-4860-8859-b36713dcbcfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" Nov 03 20:10:12.001268 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:12.001219 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1c85e9e-0e22-4860-8859-b36713dcbcfc-maas-api-tls\") pod \"maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b\" (UID: \"a1c85e9e-0e22-4860-8859-b36713dcbcfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" Nov 03 20:10:12.001308 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:12.001297 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c85e9e-0e22-4860-8859-b36713dcbcfc-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b\" (UID: \"a1c85e9e-0e22-4860-8859-b36713dcbcfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" Nov 03 20:10:12.102863 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:12.102829 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxftx\" (UniqueName: \"kubernetes.io/projected/a1c85e9e-0e22-4860-8859-b36713dcbcfc-kube-api-access-bxftx\") pod \"maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b\" (UID: \"a1c85e9e-0e22-4860-8859-b36713dcbcfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" Nov 03 20:10:12.103317 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:12.102944 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1c85e9e-0e22-4860-8859-b36713dcbcfc-metrics-certs\") pod \"maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b\" (UID: \"a1c85e9e-0e22-4860-8859-b36713dcbcfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" Nov 03 20:10:12.103317 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:12.102969 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1c85e9e-0e22-4860-8859-b36713dcbcfc-maas-api-tls\") pod \"maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b\" (UID: \"a1c85e9e-0e22-4860-8859-b36713dcbcfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" Nov 03 20:10:12.103317 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:12.102997 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c85e9e-0e22-4860-8859-b36713dcbcfc-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b\" (UID: \"a1c85e9e-0e22-4860-8859-b36713dcbcfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" Nov 03 20:10:12.103767 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:12.103736 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c85e9e-0e22-4860-8859-b36713dcbcfc-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b\" (UID: \"a1c85e9e-0e22-4860-8859-b36713dcbcfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" Nov 03 20:10:12.105887 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:12.105860 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1c85e9e-0e22-4860-8859-b36713dcbcfc-maas-api-tls\") pod \"maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b\" (UID: \"a1c85e9e-0e22-4860-8859-b36713dcbcfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" Nov 03 20:10:12.106316 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:12.106292 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1c85e9e-0e22-4860-8859-b36713dcbcfc-metrics-certs\") pod \"maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b\" (UID: \"a1c85e9e-0e22-4860-8859-b36713dcbcfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" Nov 03 20:10:12.115811 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:12.115775 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxftx\" (UniqueName: \"kubernetes.io/projected/a1c85e9e-0e22-4860-8859-b36713dcbcfc-kube-api-access-bxftx\") pod \"maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b\" (UID: \"a1c85e9e-0e22-4860-8859-b36713dcbcfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" Nov 03 20:10:12.169561 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:12.169525 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" Nov 03 20:10:12.487572 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:12.482899 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b"] Nov 03 20:10:12.669513 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:12.669474 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" event={"ID":"a1c85e9e-0e22-4860-8859-b36713dcbcfc","Type":"ContainerStarted","Data":"6e156a9bcb81201ef37915c583ca540a6168a28561a033d350aa01413f0db069"} Nov 03 20:10:12.671004 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:12.670977 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" event={"ID":"74173946-29b1-4686-985c-73d7f43b51bf","Type":"ContainerStarted","Data":"c8fdfb9b2b2428372b82e2da4824fc6b1f270acaa6237282d8f57b2f4eb46549"} Nov 03 20:10:12.671195 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:12.671017 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" event={"ID":"74173946-29b1-4686-985c-73d7f43b51bf","Type":"ContainerStarted","Data":"dd710af59ed0b444412857793e4974154f355f63a442d69383d810bc1a17e306"} Nov 03 20:10:12.671195 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:12.671100 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" Nov 03 20:10:12.694720 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:12.694663 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" podStartSLOduration=1.214582049 podStartE2EDuration="1.694644063s" totalImagesPullingTime="480.062014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:12.693747344 +0000 UTC m=+795.625614856" watchObservedRunningTime="2026-11-03 20:10:12.694644063 +0000 UTC m=+795.626511607" Nov 03 20:10:13.678106 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:13.678070 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" event={"ID":"a1c85e9e-0e22-4860-8859-b36713dcbcfc","Type":"ContainerStarted","Data":"daa8d8ad303224286c6790ba7b42461e7d32118ccd6c0846a53127f3cbbf5882"} Nov 03 20:10:13.678575 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:13.678374 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" Nov 03 20:10:13.701241 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:13.701150 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" podStartSLOduration=2.21768721 podStartE2EDuration="2.701136307s" totalImagesPullingTime="483.449097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:13.699814898 +0000 UTC m=+796.631682423" watchObservedRunningTime="2026-11-03 20:10:13.701136307 +0000 UTC m=+796.633003812" Nov 03 20:10:14.655893 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:14.655861 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" Nov 03 20:10:15.663476 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:15.663395 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" Nov 03 20:10:17.640745 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:17.640714 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp" Nov 03 20:10:18.647324 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:18.647288 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:10:18.651480 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:18.647346 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" podUID="b92265eb-94c4-42c6-b2c0-b22feab2eb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:18.651480 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:18.647373 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x" Nov 03 20:10:18.684694 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:18.684666 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" Nov 03 20:10:19.689141 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:19.689112 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" Nov 03 20:10:19.717716 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:19.717680 4918 generic.go:356] "Generic (PLEG): container finished" podID="b92265eb-94c4-42c6-b2c0-b22feab2eb07" containerID="91477df13a4f8468eabf40a759fe67b4c0a795667559c47f983c329aec919025" exitCode=0 Nov 03 20:10:19.717906 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:19.717747 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" event={"ID":"b92265eb-94c4-42c6-b2c0-b22feab2eb07","Type":"ContainerDied","Data":"91477df13a4f8468eabf40a759fe67b4c0a795667559c47f983c329aec919025"} Nov 03 20:10:19.718209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:19.718158 4918 scope.go:122] "RemoveContainer" containerID="91477df13a4f8468eabf40a759fe67b4c0a795667559c47f983c329aec919025" Nov 03 20:10:19.719701 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:19.719671 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:10:19.719809 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:19.719754 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:10:19.719809 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:19.719785 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:10:20.747445 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:20.747412 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" event={"ID":"b92265eb-94c4-42c6-b2c0-b22feab2eb07","Type":"ContainerStarted","Data":"3d8d88500ae9dedf995428040759058266fc3fbad157e04e029e80de2195f10a"} Nov 03 20:10:21.176860 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.176818 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:21.177106 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.177082 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 20:10:21.177224 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.177136 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" podUID="b92265eb-94c4-42c6-b2c0-b22feab2eb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 20:10:21.355047 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.355010 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g"] Nov 03 20:10:21.360364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.360333 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g"] Nov 03 20:10:21.360528 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.360452 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" Nov 03 20:10:21.365388 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.365359 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-3aba31-serving-cert\"" Nov 03 20:10:21.456829 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.456746 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/077732c1-4eda-4ce4-8afe-44e00e632df9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g\" (UID: \"077732c1-4eda-4ce4-8afe-44e00e632df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" Nov 03 20:10:21.457083 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.457054 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/077732c1-4eda-4ce4-8afe-44e00e632df9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g\" (UID: \"077732c1-4eda-4ce4-8afe-44e00e632df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" Nov 03 20:10:21.457217 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.457145 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/077732c1-4eda-4ce4-8afe-44e00e632df9-metrics-certs\") pod \"maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g\" (UID: \"077732c1-4eda-4ce4-8afe-44e00e632df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" Nov 03 20:10:21.457217 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.457195 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdgz5\" (UniqueName: \"kubernetes.io/projected/077732c1-4eda-4ce4-8afe-44e00e632df9-kube-api-access-bdgz5\") pod \"maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g\" (UID: \"077732c1-4eda-4ce4-8afe-44e00e632df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" Nov 03 20:10:21.557838 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.557802 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/077732c1-4eda-4ce4-8afe-44e00e632df9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g\" (UID: \"077732c1-4eda-4ce4-8afe-44e00e632df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" Nov 03 20:10:21.558049 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.557863 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/077732c1-4eda-4ce4-8afe-44e00e632df9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g\" (UID: \"077732c1-4eda-4ce4-8afe-44e00e632df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" Nov 03 20:10:21.558049 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.557908 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/077732c1-4eda-4ce4-8afe-44e00e632df9-metrics-certs\") pod \"maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g\" (UID: \"077732c1-4eda-4ce4-8afe-44e00e632df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" Nov 03 20:10:21.558049 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.557929 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdgz5\" (UniqueName: \"kubernetes.io/projected/077732c1-4eda-4ce4-8afe-44e00e632df9-kube-api-access-bdgz5\") pod \"maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g\" (UID: \"077732c1-4eda-4ce4-8afe-44e00e632df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" Nov 03 20:10:21.558669 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.558645 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/077732c1-4eda-4ce4-8afe-44e00e632df9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g\" (UID: \"077732c1-4eda-4ce4-8afe-44e00e632df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" Nov 03 20:10:21.560434 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.560411 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/077732c1-4eda-4ce4-8afe-44e00e632df9-metrics-certs\") pod \"maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g\" (UID: \"077732c1-4eda-4ce4-8afe-44e00e632df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" Nov 03 20:10:21.560526 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.560475 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/077732c1-4eda-4ce4-8afe-44e00e632df9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g\" (UID: \"077732c1-4eda-4ce4-8afe-44e00e632df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" Nov 03 20:10:21.570512 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.570487 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdgz5\" (UniqueName: \"kubernetes.io/projected/077732c1-4eda-4ce4-8afe-44e00e632df9-kube-api-access-bdgz5\") pod \"maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g\" (UID: \"077732c1-4eda-4ce4-8afe-44e00e632df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" Nov 03 20:10:21.676657 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.676627 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" Nov 03 20:10:21.975620 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:21.975590 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g"] Nov 03 20:10:21.977438 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:10:21.977404 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077732c1_4eda_4ce4_8afe_44e00e632df9.slice/crio-276b3e0163b662e01d20c51c2adbfbb518028b395756eccf5961216ce5440e56 WatchSource:0}: Error finding container 276b3e0163b662e01d20c51c2adbfbb518028b395756eccf5961216ce5440e56: Status 404 returned error can't find the container with id 276b3e0163b662e01d20c51c2adbfbb518028b395756eccf5961216ce5440e56 Nov 03 20:10:22.176890 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:22.176853 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 20:10:22.177063 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:22.176958 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" podUID="b92265eb-94c4-42c6-b2c0-b22feab2eb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 20:10:22.761272 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:22.761165 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" event={"ID":"077732c1-4eda-4ce4-8afe-44e00e632df9","Type":"ContainerStarted","Data":"bd1306b94e6ee2c499edebf6519db2670c65e6d43545c8cd9191569bd66f5a84"} Nov 03 20:10:22.761272 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:22.761244 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" event={"ID":"077732c1-4eda-4ce4-8afe-44e00e632df9","Type":"ContainerStarted","Data":"276b3e0163b662e01d20c51c2adbfbb518028b395756eccf5961216ce5440e56"} Nov 03 20:10:22.784476 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:22.784427 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" podStartSLOduration=1.390604013 podStartE2EDuration="1.784413231s" totalImagesPullingTime="393.809218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:22.781097007 +0000 UTC m=+805.712964521" watchObservedRunningTime="2026-11-03 20:10:22.784413231 +0000 UTC m=+805.716280742" Nov 03 20:10:23.176944 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:23.176911 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 20:10:23.177352 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:23.176970 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" podUID="b92265eb-94c4-42c6-b2c0-b22feab2eb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 20:10:23.766273 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:23.766242 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" Nov 03 20:10:24.176616 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:24.176576 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 20:10:24.176798 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:24.176635 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" podUID="b92265eb-94c4-42c6-b2c0-b22feab2eb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 20:10:25.177062 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:25.177024 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 20:10:25.177597 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:25.177087 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" podUID="b92265eb-94c4-42c6-b2c0-b22feab2eb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 20:10:26.773729 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:26.773696 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:26.774266 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:26.774127 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:26.789270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:26.789238 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:10:28.213147 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:10:28.213111 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:10:29.776764 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:29.776731 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" Nov 03 20:10:34.653321 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:34.653283 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:10:34.653747 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:34.653353 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" podUID="645fc172-d7a8-48a6-93ec-6c9ed0fc0304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:35.827502 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:35.827465 4918 generic.go:356] "Generic (PLEG): container finished" podID="645fc172-d7a8-48a6-93ec-6c9ed0fc0304" containerID="07c212a00a781a6e576aa7189d906443fea4dbd04b72779b4a403a8180aecb35" exitCode=0 Nov 03 20:10:35.827949 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:35.827536 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" event={"ID":"645fc172-d7a8-48a6-93ec-6c9ed0fc0304","Type":"ContainerDied","Data":"07c212a00a781a6e576aa7189d906443fea4dbd04b72779b4a403a8180aecb35"} Nov 03 20:10:35.827949 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:35.827932 4918 scope.go:122] "RemoveContainer" containerID="07c212a00a781a6e576aa7189d906443fea4dbd04b72779b4a403a8180aecb35" Nov 03 20:10:35.829305 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:35.829277 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:10:35.829372 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:35.829342 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:10:35.829409 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:35.829375 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:10:36.847982 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:36.847947 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" event={"ID":"645fc172-d7a8-48a6-93ec-6c9ed0fc0304","Type":"ContainerStarted","Data":"3402db34bde1d870cbd79c4673cff293024e825e9bf765a59df547e669031b8d"} Nov 03 20:10:37.809347 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:37.809309 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:37.809553 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:37.809503 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:10:37.809632 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:37.809570 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" podUID="645fc172-d7a8-48a6-93ec-6c9ed0fc0304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 03 20:10:38.809367 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:38.809333 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:10:38.809764 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:38.809387 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" podUID="645fc172-d7a8-48a6-93ec-6c9ed0fc0304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 03 20:10:39.809266 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:39.809227 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:10:39.809437 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:39.809298 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" podUID="645fc172-d7a8-48a6-93ec-6c9ed0fc0304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 03 20:10:40.809312 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:40.809269 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:10:40.809485 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:40.809372 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" podUID="645fc172-d7a8-48a6-93ec-6c9ed0fc0304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 03 20:10:41.809450 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:41.809404 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:10:41.809655 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:41.809461 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" podUID="645fc172-d7a8-48a6-93ec-6c9ed0fc0304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 03 20:10:42.670051 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.670019 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf"] Nov 03 20:10:42.673746 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.673728 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:42.682125 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.682062 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3aba31\"/\"openshift-service-ca.crt\"" Nov 03 20:10:42.683671 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.683647 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3aba31\"/\"default-dockercfg-rcqfz\"" Nov 03 20:10:42.683788 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.683711 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3aba31\"/\"kube-root-ca.crt\"" Nov 03 20:10:42.683788 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.683740 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3aba31\"/\"simulator-w4-3aba31-kserve-self-signed-certs\"" Nov 03 20:10:42.685980 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.685958 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf"] Nov 03 20:10:42.719764 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.719733 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-dshm\") pod \"simulator-w4-3aba31-kserve-544c766f44-2lhtf\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:42.719902 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.719782 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-home\") pod \"simulator-w4-3aba31-kserve-544c766f44-2lhtf\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:42.719902 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.719887 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-model-cache\") pod \"simulator-w4-3aba31-kserve-544c766f44-2lhtf\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:42.719978 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.719927 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-tmp-dir\") pod \"simulator-w4-3aba31-kserve-544c766f44-2lhtf\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:42.719978 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.719949 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/189c26eb-8160-4721-9775-f27379339f50-tls-certs\") pod \"simulator-w4-3aba31-kserve-544c766f44-2lhtf\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:42.720043 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.720022 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v88vt\" (UniqueName: \"kubernetes.io/projected/189c26eb-8160-4721-9775-f27379339f50-kube-api-access-v88vt\") pod \"simulator-w4-3aba31-kserve-544c766f44-2lhtf\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:42.813193 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.813153 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:42.813633 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.813614 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:42.814348 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.814332 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:10:42.821294 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.821273 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-home\") pod \"simulator-w4-3aba31-kserve-544c766f44-2lhtf\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:42.821369 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.821336 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-model-cache\") pod \"simulator-w4-3aba31-kserve-544c766f44-2lhtf\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:42.821415 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.821375 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-tmp-dir\") pod \"simulator-w4-3aba31-kserve-544c766f44-2lhtf\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:42.821415 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.821406 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/189c26eb-8160-4721-9775-f27379339f50-tls-certs\") pod \"simulator-w4-3aba31-kserve-544c766f44-2lhtf\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:42.821516 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.821461 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v88vt\" (UniqueName: \"kubernetes.io/projected/189c26eb-8160-4721-9775-f27379339f50-kube-api-access-v88vt\") pod \"simulator-w4-3aba31-kserve-544c766f44-2lhtf\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:42.821562 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.821545 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-dshm\") pod \"simulator-w4-3aba31-kserve-544c766f44-2lhtf\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:42.821765 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.821742 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-home\") pod \"simulator-w4-3aba31-kserve-544c766f44-2lhtf\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:42.821862 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.821794 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-model-cache\") pod \"simulator-w4-3aba31-kserve-544c766f44-2lhtf\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:42.821925 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.821872 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-tmp-dir\") pod \"simulator-w4-3aba31-kserve-544c766f44-2lhtf\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:42.823867 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.823845 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-dshm\") pod \"simulator-w4-3aba31-kserve-544c766f44-2lhtf\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:42.831589 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.824581 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/189c26eb-8160-4721-9775-f27379339f50-tls-certs\") pod \"simulator-w4-3aba31-kserve-544c766f44-2lhtf\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:42.834758 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.834732 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v88vt\" (UniqueName: \"kubernetes.io/projected/189c26eb-8160-4721-9775-f27379339f50-kube-api-access-v88vt\") pod \"simulator-w4-3aba31-kserve-544c766f44-2lhtf\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:42.986287 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:42.986199 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:43.317533 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:43.317496 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf"] Nov 03 20:10:43.319333 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:10:43.319301 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod189c26eb_8160_4721_9775_f27379339f50.slice/crio-0b1eab39ee4c220cbf7ec2426d8de09b4760eb0b6a0bf63385037abff597d393 WatchSource:0}: Error finding container 0b1eab39ee4c220cbf7ec2426d8de09b4760eb0b6a0bf63385037abff597d393: Status 404 returned error can't find the container with id 0b1eab39ee4c220cbf7ec2426d8de09b4760eb0b6a0bf63385037abff597d393 Nov 03 20:10:43.888571 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:43.888533 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" event={"ID":"189c26eb-8160-4721-9775-f27379339f50","Type":"ContainerStarted","Data":"4546ad176dac149c56fc9a63b563e1a918446a8108123a0e6b7723a8be3d4818"} Nov 03 20:10:43.888571 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:43.888581 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" event={"ID":"189c26eb-8160-4721-9775-f27379339f50","Type":"ContainerStarted","Data":"0b1eab39ee4c220cbf7ec2426d8de09b4760eb0b6a0bf63385037abff597d393"} Nov 03 20:10:43.908857 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:43.908809 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podStartSLOduration=1.908794987 podStartE2EDuration="1.908794987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:43.907032238 +0000 UTC m=+826.838899748" watchObservedRunningTime="2026-11-03 20:10:43.908794987 +0000 UTC m=+826.840662497" Nov 03 20:10:43.986533 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:43.986504 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:52.987238 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:52.987202 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:52.999801 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:52.999773 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:53.962641 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:53.962612 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:10:58.166217 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.166167 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw"] Nov 03 20:10:58.171884 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.171860 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:10:58.179813 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.179776 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a6f98b\"/\"default-dockercfg-7zd8f\"" Nov 03 20:10:58.180298 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.180280 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-a6f98b\"/\"openshift-service-ca.crt\"" Nov 03 20:10:58.180818 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.180796 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-a6f98b\"/\"kube-root-ca.crt\"" Nov 03 20:10:58.181026 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.181006 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a6f98b\"/\"premium-w5-a6f98b-kserve-self-signed-certs\"" Nov 03 20:10:58.182057 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.182035 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw"] Nov 03 20:10:58.211133 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:10:58.211098 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:10:58.287126 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.287077 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv4mp\" (UniqueName: \"kubernetes.io/projected/fe6c2991-de3a-4374-a5c3-63816a171924-kube-api-access-pv4mp\") pod \"premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:10:58.287126 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.287131 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-home\") pod \"premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:10:58.287350 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.287324 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-tmp-dir\") pod \"premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:10:58.287393 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.287358 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-dshm\") pod \"premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:10:58.287393 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.287386 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-model-cache\") pod \"premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:10:58.287472 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.287454 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6c2991-de3a-4374-a5c3-63816a171924-tls-certs\") pod \"premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:10:58.388267 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.388219 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv4mp\" (UniqueName: \"kubernetes.io/projected/fe6c2991-de3a-4374-a5c3-63816a171924-kube-api-access-pv4mp\") pod \"premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:10:58.388449 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.388279 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-home\") pod \"premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:10:58.388449 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.388356 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-tmp-dir\") pod \"premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:10:58.388449 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.388381 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-dshm\") pod \"premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:10:58.388449 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.388404 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-model-cache\") pod \"premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:10:58.388612 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.388452 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6c2991-de3a-4374-a5c3-63816a171924-tls-certs\") pod \"premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:10:58.388736 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.388716 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-home\") pod \"premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:10:58.388811 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.388789 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-tmp-dir\") pod \"premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:10:58.388811 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.388804 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-model-cache\") pod \"premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:10:58.390826 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.390795 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-dshm\") pod \"premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:10:58.391072 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.391053 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6c2991-de3a-4374-a5c3-63816a171924-tls-certs\") pod \"premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:10:58.400442 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.400411 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv4mp\" (UniqueName: \"kubernetes.io/projected/fe6c2991-de3a-4374-a5c3-63816a171924-kube-api-access-pv4mp\") pod \"premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:10:58.487735 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.487659 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:10:58.767332 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.767307 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw"] Nov 03 20:10:58.769140 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:10:58.769111 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe6c2991_de3a_4374_a5c3_63816a171924.slice/crio-3cc6872f8bb7f85d204b1bcb009cb306b8942f2db3b92ebd2d8359fc17699130 WatchSource:0}: Error finding container 3cc6872f8bb7f85d204b1bcb009cb306b8942f2db3b92ebd2d8359fc17699130: Status 404 returned error can't find the container with id 3cc6872f8bb7f85d204b1bcb009cb306b8942f2db3b92ebd2d8359fc17699130 Nov 03 20:10:58.992151 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.992117 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" event={"ID":"fe6c2991-de3a-4374-a5c3-63816a171924","Type":"ContainerStarted","Data":"5d56270e330d2b3763f2a63e9b385deb8a2ae8c55b77949735acc0d77fc28523"} Nov 03 20:10:58.992348 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:58.992160 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" event={"ID":"fe6c2991-de3a-4374-a5c3-63816a171924","Type":"ContainerStarted","Data":"3cc6872f8bb7f85d204b1bcb009cb306b8942f2db3b92ebd2d8359fc17699130"} Nov 03 20:10:59.011020 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:59.010969 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podStartSLOduration=1.010956155 podStartE2EDuration="1.010956155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:59.010342848 +0000 UTC m=+841.942210362" watchObservedRunningTime="2026-11-03 20:10:59.010956155 +0000 UTC m=+841.942823690" Nov 03 20:10:59.488798 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:10:59.488764 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:11:03.629881 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.629843 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66"] Nov 03 20:11:03.633805 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.633785 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:03.637641 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.637619 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-d412cf\"/\"openshift-service-ca.crt\"" Nov 03 20:11:03.638872 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.638850 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d412cf\"/\"premium-w6-d412cf-kserve-self-signed-certs\"" Nov 03 20:11:03.638965 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.638915 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d412cf\"/\"default-dockercfg-l8wbg\"" Nov 03 20:11:03.640406 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.640386 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-d412cf\"/\"kube-root-ca.crt\"" Nov 03 20:11:03.643985 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.643962 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66"] Nov 03 20:11:03.748113 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.748071 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-model-cache\") pod \"premium-w6-d412cf-kserve-567868994b-hrm66\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:03.748326 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.748119 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-home\") pod \"premium-w6-d412cf-kserve-567868994b-hrm66\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:03.748326 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.748200 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-tmp-dir\") pod \"premium-w6-d412cf-kserve-567868994b-hrm66\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:03.748451 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.748346 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bad512a6-de2f-428d-a37a-e56eb56e15f6-tls-certs\") pod \"premium-w6-d412cf-kserve-567868994b-hrm66\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:03.748510 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.748447 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t64sp\" (UniqueName: \"kubernetes.io/projected/bad512a6-de2f-428d-a37a-e56eb56e15f6-kube-api-access-t64sp\") pod \"premium-w6-d412cf-kserve-567868994b-hrm66\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:03.748510 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.748497 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-dshm\") pod \"premium-w6-d412cf-kserve-567868994b-hrm66\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:03.849643 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.849599 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bad512a6-de2f-428d-a37a-e56eb56e15f6-tls-certs\") pod \"premium-w6-d412cf-kserve-567868994b-hrm66\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:03.849829 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.849662 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t64sp\" (UniqueName: \"kubernetes.io/projected/bad512a6-de2f-428d-a37a-e56eb56e15f6-kube-api-access-t64sp\") pod \"premium-w6-d412cf-kserve-567868994b-hrm66\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:03.849829 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.849705 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-dshm\") pod \"premium-w6-d412cf-kserve-567868994b-hrm66\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:03.849829 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.849785 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-model-cache\") pod \"premium-w6-d412cf-kserve-567868994b-hrm66\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:03.849829 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.849814 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-home\") pod \"premium-w6-d412cf-kserve-567868994b-hrm66\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:03.850006 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.849844 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-tmp-dir\") pod \"premium-w6-d412cf-kserve-567868994b-hrm66\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:03.850283 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.850258 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-home\") pod \"premium-w6-d412cf-kserve-567868994b-hrm66\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:03.850371 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.850284 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-model-cache\") pod \"premium-w6-d412cf-kserve-567868994b-hrm66\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:03.850371 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.850324 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-tmp-dir\") pod \"premium-w6-d412cf-kserve-567868994b-hrm66\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:03.852276 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.852254 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-dshm\") pod \"premium-w6-d412cf-kserve-567868994b-hrm66\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:03.852700 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.852680 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bad512a6-de2f-428d-a37a-e56eb56e15f6-tls-certs\") pod \"premium-w6-d412cf-kserve-567868994b-hrm66\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:03.861839 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.861812 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t64sp\" (UniqueName: \"kubernetes.io/projected/bad512a6-de2f-428d-a37a-e56eb56e15f6-kube-api-access-t64sp\") pod \"premium-w6-d412cf-kserve-567868994b-hrm66\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:03.973169 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:03.973085 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:04.257079 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:04.257050 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66"] Nov 03 20:11:04.259379 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:11:04.259326 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbad512a6_de2f_428d_a37a_e56eb56e15f6.slice/crio-b44bda2fcc318d8691786342e7b633e5279159bea2dfcafe06306373ade94837 WatchSource:0}: Error finding container b44bda2fcc318d8691786342e7b633e5279159bea2dfcafe06306373ade94837: Status 404 returned error can't find the container with id b44bda2fcc318d8691786342e7b633e5279159bea2dfcafe06306373ade94837 Nov 03 20:11:05.032096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:05.032060 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" event={"ID":"bad512a6-de2f-428d-a37a-e56eb56e15f6","Type":"ContainerStarted","Data":"92b7691d4dadc42955585f101711b585ff3ac08b5dab1aa884aad18af3a79b7c"} Nov 03 20:11:05.032567 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:05.032105 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" event={"ID":"bad512a6-de2f-428d-a37a-e56eb56e15f6","Type":"ContainerStarted","Data":"b44bda2fcc318d8691786342e7b633e5279159bea2dfcafe06306373ade94837"} Nov 03 20:11:05.053281 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:05.053230 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podStartSLOduration=2.053214474 podStartE2EDuration="2.053214474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:05.050996467 +0000 UTC m=+847.982863981" watchObservedRunningTime="2026-11-03 20:11:05.053214474 +0000 UTC m=+847.985082022" Nov 03 20:11:05.973731 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:05.973697 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:08.488110 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:08.488027 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:11:08.500774 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:08.500747 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:11:09.069770 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:09.069739 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:11:13.973569 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:13.973532 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:13.986737 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:13.986706 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:14.097565 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:14.097530 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:11:16.409406 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.409372 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt"] Nov 03 20:11:16.413703 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.413676 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:16.422342 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.421913 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8013d9\"/\"openshift-service-ca.crt\"" Nov 03 20:11:16.422879 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.422857 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8013d9\"/\"premium-w1-8013d9-kserve-self-signed-certs\"" Nov 03 20:11:16.423002 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.422990 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8013d9\"/\"default-dockercfg-cvfv5\"" Nov 03 20:11:16.423598 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.423577 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8013d9\"/\"kube-root-ca.crt\"" Nov 03 20:11:16.425059 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.425038 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt"] Nov 03 20:11:16.490449 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.490412 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrsvz\" (UniqueName: \"kubernetes.io/projected/b14fd53e-9193-44a0-9693-446fe067a841-kube-api-access-zrsvz\") pod \"premium-w1-8013d9-kserve-84847c8cf6-j2xlt\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:16.490630 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.490465 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-tmp-dir\") pod \"premium-w1-8013d9-kserve-84847c8cf6-j2xlt\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:16.490630 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.490492 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-home\") pod \"premium-w1-8013d9-kserve-84847c8cf6-j2xlt\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:16.490700 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.490649 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-model-cache\") pod \"premium-w1-8013d9-kserve-84847c8cf6-j2xlt\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:16.490700 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.490685 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b14fd53e-9193-44a0-9693-446fe067a841-tls-certs\") pod \"premium-w1-8013d9-kserve-84847c8cf6-j2xlt\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:16.490808 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.490787 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-dshm\") pod \"premium-w1-8013d9-kserve-84847c8cf6-j2xlt\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:16.592161 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.592124 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-model-cache\") pod \"premium-w1-8013d9-kserve-84847c8cf6-j2xlt\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:16.592161 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.592160 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b14fd53e-9193-44a0-9693-446fe067a841-tls-certs\") pod \"premium-w1-8013d9-kserve-84847c8cf6-j2xlt\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:16.592408 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.592214 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-dshm\") pod \"premium-w1-8013d9-kserve-84847c8cf6-j2xlt\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:16.592408 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.592258 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrsvz\" (UniqueName: \"kubernetes.io/projected/b14fd53e-9193-44a0-9693-446fe067a841-kube-api-access-zrsvz\") pod \"premium-w1-8013d9-kserve-84847c8cf6-j2xlt\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:16.592524 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.592408 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-tmp-dir\") pod \"premium-w1-8013d9-kserve-84847c8cf6-j2xlt\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:16.592524 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.592482 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-home\") pod \"premium-w1-8013d9-kserve-84847c8cf6-j2xlt\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:16.592619 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.592572 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-model-cache\") pod \"premium-w1-8013d9-kserve-84847c8cf6-j2xlt\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:16.592717 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.592698 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-tmp-dir\") pod \"premium-w1-8013d9-kserve-84847c8cf6-j2xlt\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:16.592797 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.592776 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-home\") pod \"premium-w1-8013d9-kserve-84847c8cf6-j2xlt\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:16.594517 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.594498 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-dshm\") pod \"premium-w1-8013d9-kserve-84847c8cf6-j2xlt\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:16.594787 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.594769 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b14fd53e-9193-44a0-9693-446fe067a841-tls-certs\") pod \"premium-w1-8013d9-kserve-84847c8cf6-j2xlt\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:16.604549 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.604527 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrsvz\" (UniqueName: \"kubernetes.io/projected/b14fd53e-9193-44a0-9693-446fe067a841-kube-api-access-zrsvz\") pod \"premium-w1-8013d9-kserve-84847c8cf6-j2xlt\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:16.728579 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:16.728481 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:17.007769 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:17.007738 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt"] Nov 03 20:11:17.009250 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:11:17.009222 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb14fd53e_9193_44a0_9693_446fe067a841.slice/crio-be6ca1a9ff4f22217e1c507dc92cf0fec149ba6649ee5a0138726ef005b5fb29 WatchSource:0}: Error finding container be6ca1a9ff4f22217e1c507dc92cf0fec149ba6649ee5a0138726ef005b5fb29: Status 404 returned error can't find the container with id be6ca1a9ff4f22217e1c507dc92cf0fec149ba6649ee5a0138726ef005b5fb29 Nov 03 20:11:17.105220 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:17.105170 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" event={"ID":"b14fd53e-9193-44a0-9693-446fe067a841","Type":"ContainerStarted","Data":"74fe3f36ca790f86d20f06f6e23b3e6a09da1850873af80fd6696dfb3b3bb417"} Nov 03 20:11:17.105356 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:17.105227 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" event={"ID":"b14fd53e-9193-44a0-9693-446fe067a841","Type":"ContainerStarted","Data":"be6ca1a9ff4f22217e1c507dc92cf0fec149ba6649ee5a0138726ef005b5fb29"} Nov 03 20:11:17.124517 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:17.124462 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podStartSLOduration=1.124446804 podStartE2EDuration="1.124446804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:17.123791667 +0000 UTC m=+860.055659179" watchObservedRunningTime="2026-11-03 20:11:17.124446804 +0000 UTC m=+860.056314313" Nov 03 20:11:17.728940 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:17.728900 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:26.729440 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:26.729406 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:26.746320 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:26.746283 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:27.169734 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:27.169702 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:11:28.207246 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:11:28.207209 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:11:57.288848 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:57.288819 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:11:57.289518 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:11:57.289427 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:11:58.206951 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:11:58.206913 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:12:00.785334 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:00.785289 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd"] Nov 03 20:12:00.790388 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:00.790363 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:00.805460 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:00.805432 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-8013d9-openshift-default-dockercfg-f7g58\"" Nov 03 20:12:00.806334 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:00.806308 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd"] Nov 03 20:12:00.932281 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:00.932240 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:00.932483 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:00.932292 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:00.932483 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:00.932426 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:00.932483 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:00.932464 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:00.932618 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:00.932495 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:00.932656 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:00.932618 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:00.932708 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:00.932653 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:00.932764 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:00.932718 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:00.932844 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:00.932826 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb7k4\" (UniqueName: \"kubernetes.io/projected/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-kube-api-access-qb7k4\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:01.034447 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.034407 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb7k4\" (UniqueName: \"kubernetes.io/projected/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-kube-api-access-qb7k4\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:01.034653 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.034475 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:01.034653 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.034510 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:01.034653 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.034581 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:01.034653 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.034611 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:01.034653 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.034640 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:01.034937 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.034725 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:01.034937 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.034752 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:01.034937 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.034787 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:01.035095 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.035030 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:01.035095 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.035060 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:01.035299 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.035271 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:01.035524 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.035474 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:01.035650 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.035627 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:01.037028 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.037001 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:01.037366 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.037338 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:01.059406 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.059040 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:01.059406 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.059312 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb7k4\" (UniqueName: \"kubernetes.io/projected/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-kube-api-access-qb7k4\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:01.109002 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.108969 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:01.402337 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.402310 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd"] Nov 03 20:12:01.403764 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:12:01.403736 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ea5a18_974d_456d_88b5_e05b8cd33f9c.slice/crio-665f0fe3977d192028484f5dd4ae59c02e09fb02154363917784769a528e0dce WatchSource:0}: Error finding container 665f0fe3977d192028484f5dd4ae59c02e09fb02154363917784769a528e0dce: Status 404 returned error can't find the container with id 665f0fe3977d192028484f5dd4ae59c02e09fb02154363917784769a528e0dce Nov 03 20:12:01.406348 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.406309 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:12:01.406462 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.406392 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:12:01.406462 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:01.406441 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:12:02.389588 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:02.389555 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" event={"ID":"a7ea5a18-974d-456d-88b5-e05b8cd33f9c","Type":"ContainerStarted","Data":"f51d09be5c875de99f1b4970764e01c0c7a86249a9ec4135ad6329b200288070"} Nov 03 20:12:02.389967 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:02.389611 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" event={"ID":"a7ea5a18-974d-456d-88b5-e05b8cd33f9c","Type":"ContainerStarted","Data":"665f0fe3977d192028484f5dd4ae59c02e09fb02154363917784769a528e0dce"} Nov 03 20:12:02.483196 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:02.483118 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" podStartSLOduration=2.483099979 podStartE2EDuration="2.483099979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:02.480832936 +0000 UTC m=+905.412700446" watchObservedRunningTime="2026-11-03 20:12:02.483099979 +0000 UTC m=+905.414967492" Nov 03 20:12:03.109561 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:03.109481 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:03.111758 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:03.111432 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:12:03.111758 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:03.111500 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" podUID="a7ea5a18-974d-456d-88b5-e05b8cd33f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 20:12:04.109481 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:04.109433 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:12:04.110059 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:04.110018 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" podUID="a7ea5a18-974d-456d-88b5-e05b8cd33f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 20:12:05.154196 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:05.149320 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:12:05.154196 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:05.149402 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" podUID="a7ea5a18-974d-456d-88b5-e05b8cd33f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 20:12:06.110852 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:06.110803 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:12:06.111004 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:06.110870 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" podUID="a7ea5a18-974d-456d-88b5-e05b8cd33f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 20:12:07.109855 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:07.109813 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:12:07.110319 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:07.109886 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" podUID="a7ea5a18-974d-456d-88b5-e05b8cd33f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 20:12:08.198456 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:08.198418 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:08.198938 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:08.198904 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:08.199870 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:08.199849 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:12:10.558045 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.558002 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk"] Nov 03 20:12:10.564042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.564012 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.583151 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.578418 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk"] Nov 03 20:12:10.583151 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.579098 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-62221d-openshift-default-dockercfg-tqhkb\"" Nov 03 20:12:10.622237 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.622165 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/935c467f-7d4e-4ea6-9403-08b0079f774c-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.622431 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.622370 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.622500 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.622444 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.622552 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.622534 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.622589 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.622568 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.622644 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.622594 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.622711 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.622692 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95h7x\" (UniqueName: \"kubernetes.io/projected/935c467f-7d4e-4ea6-9403-08b0079f774c-kube-api-access-95h7x\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.622768 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.622732 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.622814 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.622789 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.723449 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.723410 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.723632 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.723472 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.723632 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.723498 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.723632 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.723514 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.723632 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.723542 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95h7x\" (UniqueName: \"kubernetes.io/projected/935c467f-7d4e-4ea6-9403-08b0079f774c-kube-api-access-95h7x\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.723632 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.723563 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.723632 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.723582 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.723632 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.723618 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/935c467f-7d4e-4ea6-9403-08b0079f774c-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.724006 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.723708 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.724006 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.723891 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.724006 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.723930 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.724006 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.723977 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.724167 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.724090 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.724495 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.724476 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/935c467f-7d4e-4ea6-9403-08b0079f774c-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.725898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.725870 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.726165 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.726144 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.735613 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.735582 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.735725 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.735664 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95h7x\" (UniqueName: \"kubernetes.io/projected/935c467f-7d4e-4ea6-9403-08b0079f774c-kube-api-access-95h7x\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:10.895234 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:10.895155 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:11.196105 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:11.196077 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk"] Nov 03 20:12:11.197600 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:12:11.197570 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935c467f_7d4e_4ea6_9403_08b0079f774c.slice/crio-ada934db5a44a8dbb45dc88b4818e2d8d7ecc466d707f32b944197b3ad8aa529 WatchSource:0}: Error finding container ada934db5a44a8dbb45dc88b4818e2d8d7ecc466d707f32b944197b3ad8aa529: Status 404 returned error can't find the container with id ada934db5a44a8dbb45dc88b4818e2d8d7ecc466d707f32b944197b3ad8aa529 Nov 03 20:12:11.200071 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:11.200036 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:12:11.200149 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:11.200107 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:12:11.200149 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:11.200137 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:12:11.483849 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:11.483761 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" event={"ID":"935c467f-7d4e-4ea6-9403-08b0079f774c","Type":"ContainerStarted","Data":"474619fc9ed4e6b491a6dcbca7b333cdf2a3a4ac18fee2b774e197be58f0b777"} Nov 03 20:12:11.483849 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:11.483822 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" event={"ID":"935c467f-7d4e-4ea6-9403-08b0079f774c","Type":"ContainerStarted","Data":"ada934db5a44a8dbb45dc88b4818e2d8d7ecc466d707f32b944197b3ad8aa529"} Nov 03 20:12:11.896036 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:11.896001 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:12.897428 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:12.897393 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:12:12.897877 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:12.897456 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" podUID="935c467f-7d4e-4ea6-9403-08b0079f774c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 20:12:13.896515 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:13.896476 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:12:13.896675 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:13.896537 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" podUID="935c467f-7d4e-4ea6-9403-08b0079f774c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 20:12:14.896642 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:14.896599 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:12:14.897315 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:14.896672 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" podUID="935c467f-7d4e-4ea6-9403-08b0079f774c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 20:12:15.896419 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:15.896379 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:12:15.896594 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:15.896440 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" podUID="935c467f-7d4e-4ea6-9403-08b0079f774c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 20:12:16.895636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:16.895605 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:12:16.896054 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:16.895655 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" podUID="935c467f-7d4e-4ea6-9403-08b0079f774c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 20:12:17.896545 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:17.896505 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:12:17.897020 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:17.896563 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" podUID="935c467f-7d4e-4ea6-9403-08b0079f774c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 20:12:18.900053 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:18.900022 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:18.900520 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:18.900505 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:18.901239 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:18.901219 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:12:18.924982 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:18.924929 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" podStartSLOduration=8.924916886 podStartE2EDuration="8.924916886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:11.512412742 +0000 UTC m=+914.444280256" watchObservedRunningTime="2026-11-03 20:12:18.924916886 +0000 UTC m=+921.856784412" Nov 03 20:12:28.211631 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:12:28.211583 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:12:42.650794 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.650707 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v"] Nov 03 20:12:42.655285 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.655262 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:42.662299 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.662274 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8013d9\"/\"distinct-w1-8013d9-kserve-self-signed-certs\"" Nov 03 20:12:42.666285 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.666264 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v"] Nov 03 20:12:42.733600 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.733559 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-tmp-dir\") pod \"distinct-w1-8013d9-kserve-5864c8c96c-qs62v\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:42.733898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.733873 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kddd7\" (UniqueName: \"kubernetes.io/projected/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-kube-api-access-kddd7\") pod \"distinct-w1-8013d9-kserve-5864c8c96c-qs62v\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:42.734040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.734025 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-model-cache\") pod \"distinct-w1-8013d9-kserve-5864c8c96c-qs62v\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:42.734151 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.734137 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-home\") pod \"distinct-w1-8013d9-kserve-5864c8c96c-qs62v\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:42.734348 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.734332 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-dshm\") pod \"distinct-w1-8013d9-kserve-5864c8c96c-qs62v\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:42.734473 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.734457 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-tls-certs\") pod \"distinct-w1-8013d9-kserve-5864c8c96c-qs62v\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:42.838562 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.836621 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kddd7\" (UniqueName: \"kubernetes.io/projected/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-kube-api-access-kddd7\") pod \"distinct-w1-8013d9-kserve-5864c8c96c-qs62v\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:42.838562 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.836681 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-model-cache\") pod \"distinct-w1-8013d9-kserve-5864c8c96c-qs62v\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:42.838562 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.836715 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-home\") pod \"distinct-w1-8013d9-kserve-5864c8c96c-qs62v\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:42.838562 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.836808 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-dshm\") pod \"distinct-w1-8013d9-kserve-5864c8c96c-qs62v\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:42.838562 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.836851 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-tls-certs\") pod \"distinct-w1-8013d9-kserve-5864c8c96c-qs62v\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:42.838562 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.836963 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-tmp-dir\") pod \"distinct-w1-8013d9-kserve-5864c8c96c-qs62v\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:42.838562 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.837438 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-tmp-dir\") pod \"distinct-w1-8013d9-kserve-5864c8c96c-qs62v\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:42.838562 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.837967 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-model-cache\") pod \"distinct-w1-8013d9-kserve-5864c8c96c-qs62v\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:42.838562 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.838218 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-home\") pod \"distinct-w1-8013d9-kserve-5864c8c96c-qs62v\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:42.881536 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.872795 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kddd7\" (UniqueName: \"kubernetes.io/projected/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-kube-api-access-kddd7\") pod \"distinct-w1-8013d9-kserve-5864c8c96c-qs62v\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:42.882753 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.882689 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-tls-certs\") pod \"distinct-w1-8013d9-kserve-5864c8c96c-qs62v\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:42.889521 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.889488 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-dshm\") pod \"distinct-w1-8013d9-kserve-5864c8c96c-qs62v\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:42.979692 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:42.979612 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:43.460245 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:43.460208 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v"] Nov 03 20:12:43.462277 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:12:43.462240 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f5ce248_0c9e_4f39_9130_1b2aad5beb65.slice/crio-0115ad0c02505ad9337dcd8e0c5e922ab41c48ecb8fe65884897a42263d19deb WatchSource:0}: Error finding container 0115ad0c02505ad9337dcd8e0c5e922ab41c48ecb8fe65884897a42263d19deb: Status 404 returned error can't find the container with id 0115ad0c02505ad9337dcd8e0c5e922ab41c48ecb8fe65884897a42263d19deb Nov 03 20:12:43.739579 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:43.739467 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" event={"ID":"6f5ce248-0c9e-4f39-9130-1b2aad5beb65","Type":"ContainerStarted","Data":"3e4f1c5f1cac945da0445311a715a360e20769378f7435fe6c6ba4cffa7fe07d"} Nov 03 20:12:43.739579 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:43.739536 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" event={"ID":"6f5ce248-0c9e-4f39-9130-1b2aad5beb65","Type":"ContainerStarted","Data":"0115ad0c02505ad9337dcd8e0c5e922ab41c48ecb8fe65884897a42263d19deb"} Nov 03 20:12:43.758954 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:43.758892 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podStartSLOduration=1.7588744649999999 podStartE2EDuration="1.758874465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:43.758491265 +0000 UTC m=+946.690358779" watchObservedRunningTime="2026-11-03 20:12:43.758874465 +0000 UTC m=+946.690741978" Nov 03 20:12:43.984207 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:43.981114 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:45.904356 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:45.904307 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g"] Nov 03 20:12:45.908828 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:45.908802 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:45.924607 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:45.924576 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g"] Nov 03 20:12:45.974596 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:45.973460 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b1b190-openshift-default-dockercfg-g8zqz\"" Nov 03 20:12:46.006133 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.006100 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.006133 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.006136 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.006346 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.006295 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.006384 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.006361 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.006418 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.006403 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.006482 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.006470 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.006554 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.006540 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ztp2\" (UniqueName: \"kubernetes.io/projected/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-kube-api-access-9ztp2\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.006589 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.006574 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.006622 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.006600 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.017485 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.017462 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc"] Nov 03 20:12:46.021760 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.021738 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.036473 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.036430 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc"] Nov 03 20:12:46.071756 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.071718 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-90ef9c-openshift-default-dockercfg-9s9f5\"" Nov 03 20:12:46.107443 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.107408 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.107622 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.107459 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz6j6\" (UniqueName: \"kubernetes.io/projected/461d7a11-e952-4fbf-a510-e1a83d824555-kube-api-access-vz6j6\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.107622 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.107491 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/461d7a11-e952-4fbf-a510-e1a83d824555-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.107622 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.107540 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.107734 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.107700 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ztp2\" (UniqueName: \"kubernetes.io/projected/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-kube-api-access-9ztp2\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.107774 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.107738 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.107774 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.107765 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.107858 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.107791 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.107858 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.107808 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.107858 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.107844 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.107960 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.107944 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.108010 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.107981 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.108046 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.108015 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.108046 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.108041 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.108155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.108050 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.108155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.108080 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.108155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.108105 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/461d7a11-e952-4fbf-a510-e1a83d824555-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.108155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.108130 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.108408 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.108167 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/461d7a11-e952-4fbf-a510-e1a83d824555-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.108408 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.108247 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.108408 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.108205 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.108408 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.108374 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.108551 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.108532 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.110247 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.110222 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.110507 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.110488 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.125525 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.125491 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.126406 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.126378 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ztp2\" (UniqueName: \"kubernetes.io/projected/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-kube-api-access-9ztp2\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.209794 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.209698 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.209794 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.209752 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.209794 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.209792 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/461d7a11-e952-4fbf-a510-e1a83d824555-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.210080 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.209827 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/461d7a11-e952-4fbf-a510-e1a83d824555-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.210080 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.209893 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz6j6\" (UniqueName: \"kubernetes.io/projected/461d7a11-e952-4fbf-a510-e1a83d824555-kube-api-access-vz6j6\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.210080 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.209918 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/461d7a11-e952-4fbf-a510-e1a83d824555-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.210080 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.210027 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.210080 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.210051 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.210080 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.210080 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.210471 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.210197 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.210471 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.210284 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.210471 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.210436 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.210698 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.210670 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/461d7a11-e952-4fbf-a510-e1a83d824555-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.210806 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.210703 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.212520 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.212500 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.212675 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.212656 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/461d7a11-e952-4fbf-a510-e1a83d824555-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.226300 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.226154 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:46.227478 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.227448 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/461d7a11-e952-4fbf-a510-e1a83d824555-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.227601 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.227486 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz6j6\" (UniqueName: \"kubernetes.io/projected/461d7a11-e952-4fbf-a510-e1a83d824555-kube-api-access-vz6j6\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.343793 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.343465 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:46.533618 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.533589 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g"] Nov 03 20:12:46.536642 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:12:46.536609 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308f3beb_2096_41e2_8216_8b6e4a6a9e5c.slice/crio-7c421d8f93938f365e1c7f1674b223e4dec0e2294c2b4d755a250a81405abd7e WatchSource:0}: Error finding container 7c421d8f93938f365e1c7f1674b223e4dec0e2294c2b4d755a250a81405abd7e: Status 404 returned error can't find the container with id 7c421d8f93938f365e1c7f1674b223e4dec0e2294c2b4d755a250a81405abd7e Nov 03 20:12:46.540331 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.540296 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:12:46.540422 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.540367 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:12:46.540422 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.540404 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:12:46.643192 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.643137 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc"] Nov 03 20:12:46.647680 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:12:46.647645 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461d7a11_e952_4fbf_a510_e1a83d824555.slice/crio-3dee98bbc9ab322969bc4b2e2b57c3ca3be50710088f875c28f18c83405a8de4 WatchSource:0}: Error finding container 3dee98bbc9ab322969bc4b2e2b57c3ca3be50710088f875c28f18c83405a8de4: Status 404 returned error can't find the container with id 3dee98bbc9ab322969bc4b2e2b57c3ca3be50710088f875c28f18c83405a8de4 Nov 03 20:12:46.650642 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.650599 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:12:46.650736 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.650695 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:12:46.650779 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.650745 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:12:46.773597 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.773557 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" event={"ID":"308f3beb-2096-41e2-8216-8b6e4a6a9e5c","Type":"ContainerStarted","Data":"dfb0f22bfa2516b278f413609d8c451fa954fccce3568fd87cf2072bb14f21fc"} Nov 03 20:12:46.773848 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.773818 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" event={"ID":"308f3beb-2096-41e2-8216-8b6e4a6a9e5c","Type":"ContainerStarted","Data":"7c421d8f93938f365e1c7f1674b223e4dec0e2294c2b4d755a250a81405abd7e"} Nov 03 20:12:46.776033 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.776005 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" event={"ID":"461d7a11-e952-4fbf-a510-e1a83d824555","Type":"ContainerStarted","Data":"74df9c2e396000e1ae488117f0dce925935824ebfc85537b752c7f7738f3d7e0"} Nov 03 20:12:46.776169 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.776056 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" event={"ID":"461d7a11-e952-4fbf-a510-e1a83d824555","Type":"ContainerStarted","Data":"3dee98bbc9ab322969bc4b2e2b57c3ca3be50710088f875c28f18c83405a8de4"} Nov 03 20:12:46.806670 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.806567 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" podStartSLOduration=1.8065475439999998 podStartE2EDuration="1.806547544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:46.800138861 +0000 UTC m=+949.732006370" watchObservedRunningTime="2026-11-03 20:12:46.806547544 +0000 UTC m=+949.738415058" Nov 03 20:12:46.825330 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:46.825268 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" podStartSLOduration=1.825248854 podStartE2EDuration="1.825248854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:46.822543906 +0000 UTC m=+949.754411430" watchObservedRunningTime="2026-11-03 20:12:46.825248854 +0000 UTC m=+949.757116366" Nov 03 20:12:47.227544 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:47.227512 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:47.228618 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:47.228597 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:12:47.228678 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:47.228652 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" podUID="308f3beb-2096-41e2-8216-8b6e4a6a9e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 20:12:47.345463 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:47.345422 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:47.347348 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:47.347301 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:12:47.347654 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:47.347593 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" podUID="461d7a11-e952-4fbf-a510-e1a83d824555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 20:12:48.232203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:48.230691 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:12:48.232203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:48.230758 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" podUID="308f3beb-2096-41e2-8216-8b6e4a6a9e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 20:12:48.347207 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:48.346606 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:12:48.347207 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:48.346669 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" podUID="461d7a11-e952-4fbf-a510-e1a83d824555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 20:12:49.242202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.241638 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:12:49.242202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.241720 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" podUID="308f3beb-2096-41e2-8216-8b6e4a6a9e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 20:12:49.356202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.349391 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:12:49.356202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.349457 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" podUID="461d7a11-e952-4fbf-a510-e1a83d824555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 20:12:49.784800 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.784765 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg"] Nov 03 20:12:49.788957 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.788934 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:49.801550 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.798744 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3aba31\"/\"distinct-w4-3aba31-kserve-self-signed-certs\"" Nov 03 20:12:49.801639 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.801615 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg"] Nov 03 20:12:49.889263 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.889209 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/812fc025-3310-47d1-80dd-72b73ef4494e-tls-certs\") pod \"distinct-w4-3aba31-kserve-79f5c97c44-hvmqg\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:49.889467 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.889287 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-model-cache\") pod \"distinct-w4-3aba31-kserve-79f5c97c44-hvmqg\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:49.889467 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.889381 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w77bd\" (UniqueName: \"kubernetes.io/projected/812fc025-3310-47d1-80dd-72b73ef4494e-kube-api-access-w77bd\") pod \"distinct-w4-3aba31-kserve-79f5c97c44-hvmqg\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:49.889467 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.889411 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-dshm\") pod \"distinct-w4-3aba31-kserve-79f5c97c44-hvmqg\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:49.889638 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.889500 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-home\") pod \"distinct-w4-3aba31-kserve-79f5c97c44-hvmqg\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:49.889638 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.889596 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-tmp-dir\") pod \"distinct-w4-3aba31-kserve-79f5c97c44-hvmqg\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:49.991092 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.991054 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-home\") pod \"distinct-w4-3aba31-kserve-79f5c97c44-hvmqg\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:49.991452 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.991432 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-tmp-dir\") pod \"distinct-w4-3aba31-kserve-79f5c97c44-hvmqg\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:49.991612 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.991599 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/812fc025-3310-47d1-80dd-72b73ef4494e-tls-certs\") pod \"distinct-w4-3aba31-kserve-79f5c97c44-hvmqg\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:49.991726 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.991714 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-model-cache\") pod \"distinct-w4-3aba31-kserve-79f5c97c44-hvmqg\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:49.991891 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.991875 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w77bd\" (UniqueName: \"kubernetes.io/projected/812fc025-3310-47d1-80dd-72b73ef4494e-kube-api-access-w77bd\") pod \"distinct-w4-3aba31-kserve-79f5c97c44-hvmqg\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:49.991986 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.991970 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-dshm\") pod \"distinct-w4-3aba31-kserve-79f5c97c44-hvmqg\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:49.996538 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.996511 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-model-cache\") pod \"distinct-w4-3aba31-kserve-79f5c97c44-hvmqg\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:49.997006 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.996979 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-home\") pod \"distinct-w4-3aba31-kserve-79f5c97c44-hvmqg\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:49.997193 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:49.997148 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-tmp-dir\") pod \"distinct-w4-3aba31-kserve-79f5c97c44-hvmqg\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:50.002437 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:50.002401 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-dshm\") pod \"distinct-w4-3aba31-kserve-79f5c97c44-hvmqg\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:50.002685 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:50.002640 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/812fc025-3310-47d1-80dd-72b73ef4494e-tls-certs\") pod \"distinct-w4-3aba31-kserve-79f5c97c44-hvmqg\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:50.009138 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:50.009027 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w77bd\" (UniqueName: \"kubernetes.io/projected/812fc025-3310-47d1-80dd-72b73ef4494e-kube-api-access-w77bd\") pod \"distinct-w4-3aba31-kserve-79f5c97c44-hvmqg\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:50.107994 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:50.107955 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:50.244201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:50.243618 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:12:50.244201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:50.243708 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" podUID="308f3beb-2096-41e2-8216-8b6e4a6a9e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 20:12:50.353202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:50.345678 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:12:50.353202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:50.345760 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" podUID="461d7a11-e952-4fbf-a510-e1a83d824555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 20:12:50.475351 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:50.475317 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg"] Nov 03 20:12:50.477404 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:12:50.477372 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod812fc025_3310_47d1_80dd_72b73ef4494e.slice/crio-d2f205963a6c37eb686f7b02efa7118d2afbff09c53f91eb42675a86fe9481d5 WatchSource:0}: Error finding container d2f205963a6c37eb686f7b02efa7118d2afbff09c53f91eb42675a86fe9481d5: Status 404 returned error can't find the container with id d2f205963a6c37eb686f7b02efa7118d2afbff09c53f91eb42675a86fe9481d5 Nov 03 20:12:50.803227 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:50.803127 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" event={"ID":"812fc025-3310-47d1-80dd-72b73ef4494e","Type":"ContainerStarted","Data":"a147f149cc7cc82ba8a418b1de868678afa352b3645d18484e22d155fc7f6e35"} Nov 03 20:12:50.803227 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:50.803166 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" event={"ID":"812fc025-3310-47d1-80dd-72b73ef4494e","Type":"ContainerStarted","Data":"d2f205963a6c37eb686f7b02efa7118d2afbff09c53f91eb42675a86fe9481d5"} Nov 03 20:12:50.840363 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:50.840298 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podStartSLOduration=1.840276239 podStartE2EDuration="1.840276239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:50.839592852 +0000 UTC m=+953.771460367" watchObservedRunningTime="2026-11-03 20:12:50.840276239 +0000 UTC m=+953.772143752" Nov 03 20:12:51.108838 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:51.108801 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:12:51.229642 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:51.229604 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:12:51.229919 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:51.229888 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" podUID="308f3beb-2096-41e2-8216-8b6e4a6a9e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 20:12:51.361256 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:51.351740 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:12:51.361256 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:51.351801 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" podUID="461d7a11-e952-4fbf-a510-e1a83d824555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 20:12:52.228695 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:52.228655 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:12:52.229018 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:52.228972 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" podUID="308f3beb-2096-41e2-8216-8b6e4a6a9e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 20:12:52.346854 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:52.346807 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:12:52.347142 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:52.347114 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" podUID="461d7a11-e952-4fbf-a510-e1a83d824555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 20:12:52.981367 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:52.981332 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:53.000380 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:53.000350 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:53.227823 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:53.227780 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:12:53.227961 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:53.227845 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" podUID="308f3beb-2096-41e2-8216-8b6e4a6a9e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 20:12:53.344644 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:53.344605 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 20:12:53.344829 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:53.344664 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" podUID="461d7a11-e952-4fbf-a510-e1a83d824555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 20:12:53.831710 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:53.831680 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:12:54.233109 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:54.233014 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:54.233713 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:54.233516 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:54.234394 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:54.234371 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:12:54.350676 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:54.350640 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:54.352440 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:54.352415 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:54.353150 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:12:54.353130 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:12:58.207400 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:12:58.207362 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:13:00.116612 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:00.116527 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:13:00.155496 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:00.155458 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:13:00.789201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:00.788681 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48"] Nov 03 20:13:00.803200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:00.799026 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:00.821200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:00.807552 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-d412cf-openshift-default-dockercfg-qrnd5\"" Nov 03 20:13:00.827317 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:00.827287 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48"] Nov 03 20:13:00.863042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:00.862512 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0553330c-c499-46c7-a5c4-0d4465e2d817-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:00.863042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:00.862570 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:00.863042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:00.862606 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:00.863042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:00.862699 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:00.863042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:00.862728 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wqwb\" (UniqueName: \"kubernetes.io/projected/0553330c-c499-46c7-a5c4-0d4465e2d817-kube-api-access-7wqwb\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:00.863042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:00.862763 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:00.863042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:00.862790 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:00.863042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:00.862888 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:00.863042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:00.862929 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:00.967554 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:00.967524 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:00.967848 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:00.967809 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wqwb\" (UniqueName: \"kubernetes.io/projected/0553330c-c499-46c7-a5c4-0d4465e2d817-kube-api-access-7wqwb\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:00.989209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:00.968007 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:00.989576 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:00.989539 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:01.041264 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:01.015519 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:01.041264 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:01.015596 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:01.041264 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:01.015693 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0553330c-c499-46c7-a5c4-0d4465e2d817-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:01.041264 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:01.015738 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:01.041264 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:01.015775 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:01.041264 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:01.016381 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:01.041264 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:01.030004 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:01.041264 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:01.030705 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0553330c-c499-46c7-a5c4-0d4465e2d817-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:01.041264 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:01.030950 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:01.045523 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:01.045490 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:01.054212 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:01.049868 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:01.092215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:01.091704 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:13:01.093091 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:01.092905 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:01.108301 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:01.093782 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:01.349203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:01.348080 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wqwb\" (UniqueName: \"kubernetes.io/projected/0553330c-c499-46c7-a5c4-0d4465e2d817-kube-api-access-7wqwb\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:01.453831 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:01.453795 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:02.000716 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:02.000625 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48"] Nov 03 20:13:02.007470 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:13:02.007438 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0553330c_c499_46c7_a5c4_0d4465e2d817.slice/crio-da3a8a738e1edf2688ec993c5d0de6931c6c2438f49985da7051087988337c63 WatchSource:0}: Error finding container da3a8a738e1edf2688ec993c5d0de6931c6c2438f49985da7051087988337c63: Status 404 returned error can't find the container with id da3a8a738e1edf2688ec993c5d0de6931c6c2438f49985da7051087988337c63 Nov 03 20:13:02.010340 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:02.010307 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:02.010427 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:02.010375 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:02.010427 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:02.010404 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:02.997722 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:02.997673 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" event={"ID":"0553330c-c499-46c7-a5c4-0d4465e2d817","Type":"ContainerStarted","Data":"e2a4aff77761a7204bf48598e9d52a987035b53613f074cd4e46bdeb7f23bb44"} Nov 03 20:13:02.998215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:02.997740 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" event={"ID":"0553330c-c499-46c7-a5c4-0d4465e2d817","Type":"ContainerStarted","Data":"da3a8a738e1edf2688ec993c5d0de6931c6c2438f49985da7051087988337c63"} Nov 03 20:13:03.033874 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:03.033806 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" podStartSLOduration=3.033782934 podStartE2EDuration="3.033782934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:03.028112586 +0000 UTC m=+965.959980099" watchObservedRunningTime="2026-11-03 20:13:03.033782934 +0000 UTC m=+965.965650449" Nov 03 20:13:03.463360 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:03.463317 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:03.465008 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:03.464975 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:13:03.465137 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:03.465030 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" podUID="0553330c-c499-46c7-a5c4-0d4465e2d817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 20:13:04.462952 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:04.462916 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:13:04.463359 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:04.462978 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" podUID="0553330c-c499-46c7-a5c4-0d4465e2d817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 20:13:04.894261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:04.894223 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj"] Nov 03 20:13:04.898498 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:04.898480 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:04.902344 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:04.902318 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 20:13:04.902344 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:04.902322 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 20:13:04.902485 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:04.902361 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 20:13:04.902485 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:04.902410 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xj9dq\"" Nov 03 20:13:04.910797 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:04.910772 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj"] Nov 03 20:13:04.976647 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:04.976610 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:04.976909 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:04.976889 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:04.977072 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:04.977058 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:04.977304 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:04.977283 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:04.977511 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:04.977497 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:04.977696 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:04.977681 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll64r\" (UniqueName: \"kubernetes.io/projected/673a0598-869d-446b-af9c-6a2e7b0636aa-kube-api-access-ll64r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:04.977835 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:04.977823 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/673a0598-869d-446b-af9c-6a2e7b0636aa-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:05.079159 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:05.079121 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:05.079342 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:05.079204 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:05.079342 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:05.079247 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:05.079519 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:05.079494 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:05.079658 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:05.079637 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:05.079725 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:05.079708 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll64r\" (UniqueName: \"kubernetes.io/projected/673a0598-869d-446b-af9c-6a2e7b0636aa-kube-api-access-ll64r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:05.079784 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:05.079756 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/673a0598-869d-446b-af9c-6a2e7b0636aa-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:05.080258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:05.080233 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:05.081775 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:05.081754 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:05.081900 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:05.081846 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:05.082021 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:05.082004 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:05.082113 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:05.082096 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/673a0598-869d-446b-af9c-6a2e7b0636aa-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:05.102601 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:05.102574 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:05.102817 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:05.102797 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll64r\" (UniqueName: \"kubernetes.io/projected/673a0598-869d-446b-af9c-6a2e7b0636aa-kube-api-access-ll64r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s4brj\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:05.211778 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:05.211689 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:05.463943 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:05.463850 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:13:05.463943 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:05.463921 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" podUID="0553330c-c499-46c7-a5c4-0d4465e2d817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 20:13:05.521368 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:05.521264 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj"] Nov 03 20:13:05.547206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:05.547135 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:05.547329 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:05.547235 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:06.021203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:06.018320 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" event={"ID":"673a0598-869d-446b-af9c-6a2e7b0636aa","Type":"ContainerStarted","Data":"72aa70dbdc42d685768cc6b48347a0f1306dc2227cd3eca79f4a7f0f5718a1a3"} Nov 03 20:13:06.021203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:06.018388 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" event={"ID":"673a0598-869d-446b-af9c-6a2e7b0636aa","Type":"ContainerStarted","Data":"447c5715205508fca086e04b94c4682dab649360214a16b415942e3d805341f4"} Nov 03 20:13:06.021203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:06.019614 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:06.034225 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:06.023247 4918 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-s4brj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:13:06.034225 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:06.023298 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" podUID="673a0598-869d-446b-af9c-6a2e7b0636aa" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:13:06.059223 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:06.058257 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" podStartSLOduration=2.05824144 podStartE2EDuration="2.05824144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:06.04575478 +0000 UTC m=+968.977622291" watchObservedRunningTime="2026-11-03 20:13:06.05824144 +0000 UTC m=+968.990108951" Nov 03 20:13:06.464215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:06.463671 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:13:06.464215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:06.463758 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" podUID="0553330c-c499-46c7-a5c4-0d4465e2d817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 20:13:07.039201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:07.032737 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:13:07.463158 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:07.463121 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:13:07.463360 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:07.463206 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" podUID="0553330c-c499-46c7-a5c4-0d4465e2d817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 20:13:08.719096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:08.719046 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:08.719755 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:08.719472 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:08.720430 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:08.720411 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:13:15.942104 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:15.941897 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz"] Nov 03 20:13:15.947616 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:15.947146 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:15.995686 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:15.995603 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz"] Nov 03 20:13:16.000017 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:15.999982 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7"] Nov 03 20:13:16.005440 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.005169 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.012258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.012228 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7"] Nov 03 20:13:16.067117 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.067080 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.067318 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.067149 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.067318 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.067228 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.067318 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.067256 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.067318 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.067297 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.067553 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.067322 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.067553 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.067358 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf9cn\" (UniqueName: \"kubernetes.io/projected/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-kube-api-access-hf9cn\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.067553 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.067403 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.067553 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.067435 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.067553 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.067473 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.067802 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.067564 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.067802 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.067603 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.067802 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.067626 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1759364-4aca-4b1d-9f83-a81bc5829924-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.067802 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.067660 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.067802 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.067718 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl6mr\" (UniqueName: \"kubernetes.io/projected/c1759364-4aca-4b1d-9f83-a81bc5829924-kube-api-access-vl6mr\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.067802 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.067742 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.067802 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.067770 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.068152 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.067872 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.169170 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.169131 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.169397 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.169209 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.169397 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.169251 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf9cn\" (UniqueName: \"kubernetes.io/projected/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-kube-api-access-hf9cn\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.169397 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.169298 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.169397 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.169329 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.169633 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.169367 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.169633 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.169599 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.169753 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.169676 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.169753 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.169684 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.169753 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.169742 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.169901 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.169771 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1759364-4aca-4b1d-9f83-a81bc5829924-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.169901 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.169883 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.169998 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.169894 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.169998 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.169981 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl6mr\" (UniqueName: \"kubernetes.io/projected/c1759364-4aca-4b1d-9f83-a81bc5829924-kube-api-access-vl6mr\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.170096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.170010 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.170096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.170040 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.170205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.170104 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.170205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.170136 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.170205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.170137 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.170363 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.170260 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.170363 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.170317 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.170363 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.170345 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.170508 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.170369 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.170508 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.170437 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.170508 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.170446 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1759364-4aca-4b1d-9f83-a81bc5829924-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.170721 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.170698 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.170778 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.170748 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.171016 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.170994 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.172158 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.172134 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.172289 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.172157 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.172828 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.172794 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.173155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.173138 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.181637 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.181609 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf9cn\" (UniqueName: \"kubernetes.io/projected/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-kube-api-access-hf9cn\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.182038 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.182013 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.182340 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.182321 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl6mr\" (UniqueName: \"kubernetes.io/projected/c1759364-4aca-4b1d-9f83-a81bc5829924-kube-api-access-vl6mr\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.182427 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.182374 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.292248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.292156 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:16.336047 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.336014 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:16.577172 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.577141 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz"] Nov 03 20:13:16.578307 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:13:16.578282 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5120ad5e_81f7_43c1_9e1d_f0407339bb0c.slice/crio-fffeb42847a93d4686cc5cfe38a8a5009d4291d8ac343a2d3a794ac1ff19ff59 WatchSource:0}: Error finding container fffeb42847a93d4686cc5cfe38a8a5009d4291d8ac343a2d3a794ac1ff19ff59: Status 404 returned error can't find the container with id fffeb42847a93d4686cc5cfe38a8a5009d4291d8ac343a2d3a794ac1ff19ff59 Nov 03 20:13:16.580721 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.580684 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:16.580828 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.580766 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:16.580828 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.580810 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:16.641353 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.641327 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7"] Nov 03 20:13:16.642990 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:13:16.642963 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1759364_4aca_4b1d_9f83_a81bc5829924.slice/crio-ada96d27dc9c960cebd27a4c47e34fdfb8a2b9b3b3cc423a790e3d0ac443e273 WatchSource:0}: Error finding container ada96d27dc9c960cebd27a4c47e34fdfb8a2b9b3b3cc423a790e3d0ac443e273: Status 404 returned error can't find the container with id ada96d27dc9c960cebd27a4c47e34fdfb8a2b9b3b3cc423a790e3d0ac443e273 Nov 03 20:13:16.645730 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.645689 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:16.645804 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.645774 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:16.645856 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:16.645815 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:17.118042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:17.118000 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" event={"ID":"5120ad5e-81f7-43c1-9e1d-f0407339bb0c","Type":"ContainerStarted","Data":"1aba7bd51ef0926c55f9b694d24bb37be6e60f8f1a23ee86900ff18f22fe2369"} Nov 03 20:13:17.118537 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:17.118061 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" event={"ID":"5120ad5e-81f7-43c1-9e1d-f0407339bb0c","Type":"ContainerStarted","Data":"fffeb42847a93d4686cc5cfe38a8a5009d4291d8ac343a2d3a794ac1ff19ff59"} Nov 03 20:13:17.119828 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:17.119802 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" event={"ID":"c1759364-4aca-4b1d-9f83-a81bc5829924","Type":"ContainerStarted","Data":"7c9a6f2352fcd5e02018c9067261dabe414841469bf1f0539ffd722c1fe5a231"} Nov 03 20:13:17.119951 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:17.119845 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" event={"ID":"c1759364-4aca-4b1d-9f83-a81bc5829924","Type":"ContainerStarted","Data":"ada96d27dc9c960cebd27a4c47e34fdfb8a2b9b3b3cc423a790e3d0ac443e273"} Nov 03 20:13:17.145263 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:17.145154 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" podStartSLOduration=2.145142967 podStartE2EDuration="2.145142967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:17.143792318 +0000 UTC m=+980.075659828" watchObservedRunningTime="2026-11-03 20:13:17.145142967 +0000 UTC m=+980.077010478" Nov 03 20:13:17.175364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:17.175305 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" podStartSLOduration=2.175288571 podStartE2EDuration="2.175288571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:17.17181345 +0000 UTC m=+980.103680996" watchObservedRunningTime="2026-11-03 20:13:17.175288571 +0000 UTC m=+980.107156084" Nov 03 20:13:17.293206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:17.293137 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:17.294698 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:17.294667 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:13:17.294859 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:17.294727 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" podUID="5120ad5e-81f7-43c1-9e1d-f0407339bb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 20:13:17.343835 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:17.343748 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:17.345278 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:17.345250 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 20:13:17.345395 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:17.345301 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" podUID="c1759364-4aca-4b1d-9f83-a81bc5829924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 20:13:18.292814 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:18.292772 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:13:18.293331 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:18.292831 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" podUID="5120ad5e-81f7-43c1-9e1d-f0407339bb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 20:13:18.353203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:18.350207 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 20:13:18.353203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:18.350267 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" podUID="c1759364-4aca-4b1d-9f83-a81bc5829924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 20:13:19.293540 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:19.293511 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:13:19.294021 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:19.293997 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" podUID="5120ad5e-81f7-43c1-9e1d-f0407339bb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 20:13:19.362203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:19.361316 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 20:13:19.362203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:19.361384 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" podUID="c1759364-4aca-4b1d-9f83-a81bc5829924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 20:13:20.292911 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:20.292867 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:13:20.293118 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:20.292934 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" podUID="5120ad5e-81f7-43c1-9e1d-f0407339bb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 20:13:20.346645 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:20.346606 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 20:13:20.347296 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:20.347263 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" podUID="c1759364-4aca-4b1d-9f83-a81bc5829924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 20:13:21.297749 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:21.297660 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:13:21.297749 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:21.297730 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" podUID="5120ad5e-81f7-43c1-9e1d-f0407339bb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 20:13:21.360216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:21.356567 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 20:13:21.369827 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:21.360238 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" podUID="c1759364-4aca-4b1d-9f83-a81bc5829924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 20:13:22.292633 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:22.292598 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:13:22.292802 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:22.292652 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" podUID="5120ad5e-81f7-43c1-9e1d-f0407339bb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 20:13:22.343202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:22.342563 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 20:13:22.343202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:22.342629 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" podUID="c1759364-4aca-4b1d-9f83-a81bc5829924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 20:13:23.292851 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:23.292806 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:13:23.293321 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:23.292882 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" podUID="5120ad5e-81f7-43c1-9e1d-f0407339bb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 20:13:23.336794 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:23.336756 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 20:13:23.336966 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:23.336822 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" podUID="c1759364-4aca-4b1d-9f83-a81bc5829924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 20:13:24.296862 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:24.296830 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:24.297357 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:24.297333 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:24.298037 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:24.298017 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:13:24.341614 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:24.341583 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:24.341949 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:24.341926 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:24.342738 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:24.342715 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:13:28.207823 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:13:28.207783 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:13:31.004270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.004235 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd"] Nov 03 20:13:31.032778 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.032738 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.065382 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.065348 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v"] Nov 03 20:13:31.070614 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.070580 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.076208 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.071050 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd"] Nov 03 20:13:31.078376 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.078346 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v"] Nov 03 20:13:31.120673 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.120635 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.120852 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.120679 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.120852 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.120734 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.120852 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.120805 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hzr2\" (UniqueName: \"kubernetes.io/projected/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-kube-api-access-4hzr2\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.120997 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.120951 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.121043 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.121012 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.121093 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.121041 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.121134 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.121091 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.121166 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.121132 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68ft7\" (UniqueName: \"kubernetes.io/projected/eba28d4c-a907-4b6f-8d20-0a1baade4c58-kube-api-access-68ft7\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.121247 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.121233 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.121283 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.121258 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.121346 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.121333 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.121402 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.121359 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.121402 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.121393 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.121469 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.121413 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.121469 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.121456 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.121536 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.121477 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.121536 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.121506 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.200163 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.200132 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg"] Nov 03 20:13:31.206088 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.206057 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.231393 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.231354 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.231679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.231657 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.231821 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.231805 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.231968 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.231950 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hzr2\" (UniqueName: \"kubernetes.io/projected/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-kube-api-access-4hzr2\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.232351 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.232146 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.232958 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.232935 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.233113 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.233095 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.233247 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.233228 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.233391 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.233375 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68ft7\" (UniqueName: \"kubernetes.io/projected/eba28d4c-a907-4b6f-8d20-0a1baade4c58-kube-api-access-68ft7\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.233522 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.233506 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.233645 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.233630 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.233813 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.233795 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.233930 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.233915 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.234059 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.234044 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.234197 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.234162 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.234317 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.234299 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.234423 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.234309 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.234537 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.234522 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.234656 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.234641 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.234971 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.234947 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.235321 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.235088 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.235503 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.235479 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.235841 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.235820 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.236167 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.233243 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.236320 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.233334 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.236437 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.236413 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.236783 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.236763 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.239924 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.238730 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.239924 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.239844 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.240731 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.240703 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.241255 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.241230 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.243983 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.243959 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.245083 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.245059 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg"] Nov 03 20:13:31.246113 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.246089 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.246765 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.246739 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hzr2\" (UniqueName: \"kubernetes.io/projected/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-kube-api-access-4hzr2\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.247987 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.247861 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68ft7\" (UniqueName: \"kubernetes.io/projected/eba28d4c-a907-4b6f-8d20-0a1baade4c58-kube-api-access-68ft7\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.248509 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.248484 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.335549 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.335519 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.335731 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.335556 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.335778 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.335760 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.335841 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.335824 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.335984 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.335921 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.336049 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.336031 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rzgk\" (UniqueName: \"kubernetes.io/projected/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-kube-api-access-9rzgk\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.336116 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.336102 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.336267 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.336251 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.336318 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.336301 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.383571 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.383539 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:31.390612 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.390586 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:31.438109 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.438072 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.438335 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.438137 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.438335 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.438260 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.438335 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.438293 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.438519 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.438393 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.438519 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.438443 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.438519 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.438486 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.438670 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.438532 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rzgk\" (UniqueName: \"kubernetes.io/projected/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-kube-api-access-9rzgk\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.438670 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.438604 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.438993 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.438967 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.439321 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.439294 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.439730 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.439703 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.440745 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.440200 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.440745 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.440406 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.441869 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.441831 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.444600 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.444544 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.459278 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.458615 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.459278 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.458823 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rzgk\" (UniqueName: \"kubernetes.io/projected/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-kube-api-access-9rzgk\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.576854 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.576822 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:31.690961 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:13:31.690928 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75fc50e4_2ec7_4e3f_bf44_0572e998c25d.slice/crio-d6b833cf8f0123bac318021995480a161b5730b0950f0875a18d11a6eb22cef6 WatchSource:0}: Error finding container d6b833cf8f0123bac318021995480a161b5730b0950f0875a18d11a6eb22cef6: Status 404 returned error can't find the container with id d6b833cf8f0123bac318021995480a161b5730b0950f0875a18d11a6eb22cef6 Nov 03 20:13:31.691671 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.691656 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd"] Nov 03 20:13:31.694141 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.694046 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:31.694315 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.694138 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:31.694315 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.694221 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:31.695247 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:13:31.695224 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba28d4c_a907_4b6f_8d20_0a1baade4c58.slice/crio-0e47afcf75d86241e54ed9cc1aa4257a853d529832eacff985e69a31636b6bdc WatchSource:0}: Error finding container 0e47afcf75d86241e54ed9cc1aa4257a853d529832eacff985e69a31636b6bdc: Status 404 returned error can't find the container with id 0e47afcf75d86241e54ed9cc1aa4257a853d529832eacff985e69a31636b6bdc Nov 03 20:13:31.696366 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.696264 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v"] Nov 03 20:13:31.697329 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.697300 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:31.697396 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.697376 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:31.697443 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.697428 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:31.870970 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.870854 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg"] Nov 03 20:13:31.891627 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:13:31.891594 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f6c399_4702_41b4_b4f4_8920d34d7cfb.slice/crio-595576f247226938f4db15024add9d7327401ca5492fa0734a717acf060ea3db WatchSource:0}: Error finding container 595576f247226938f4db15024add9d7327401ca5492fa0734a717acf060ea3db: Status 404 returned error can't find the container with id 595576f247226938f4db15024add9d7327401ca5492fa0734a717acf060ea3db Nov 03 20:13:31.898636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.896787 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:31.898636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.896895 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:31.898636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:31.896945 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:32.272754 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:32.272668 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" event={"ID":"eba28d4c-a907-4b6f-8d20-0a1baade4c58","Type":"ContainerStarted","Data":"d16e485a9994e28ac70118f08689a0b64a93b4584d763bac507797d5e86a77b5"} Nov 03 20:13:32.273293 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:32.273262 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" event={"ID":"eba28d4c-a907-4b6f-8d20-0a1baade4c58","Type":"ContainerStarted","Data":"0e47afcf75d86241e54ed9cc1aa4257a853d529832eacff985e69a31636b6bdc"} Nov 03 20:13:32.274964 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:32.274929 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" event={"ID":"75fc50e4-2ec7-4e3f-bf44-0572e998c25d","Type":"ContainerStarted","Data":"b6960ce300cc92c859d71a6a21876ae2befc85c4fc0d0df757521511f8c5c36d"} Nov 03 20:13:32.275077 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:32.274994 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" event={"ID":"75fc50e4-2ec7-4e3f-bf44-0572e998c25d","Type":"ContainerStarted","Data":"d6b833cf8f0123bac318021995480a161b5730b0950f0875a18d11a6eb22cef6"} Nov 03 20:13:32.276883 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:32.276862 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" event={"ID":"f6f6c399-4702-41b4-b4f4-8920d34d7cfb","Type":"ContainerStarted","Data":"deced59ab6e154d65b4040443148f04c9d4b7283c67bf40ac3524e4053b9dce0"} Nov 03 20:13:32.276993 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:32.276980 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" event={"ID":"f6f6c399-4702-41b4-b4f4-8920d34d7cfb","Type":"ContainerStarted","Data":"595576f247226938f4db15024add9d7327401ca5492fa0734a717acf060ea3db"} Nov 03 20:13:32.296714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:32.296655 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" podStartSLOduration=2.296637769 podStartE2EDuration="2.296637769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:32.295532629 +0000 UTC m=+995.227400138" watchObservedRunningTime="2026-11-03 20:13:32.296637769 +0000 UTC m=+995.228505282" Nov 03 20:13:32.325881 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:32.325826 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" podStartSLOduration=1.325812618 podStartE2EDuration="1.325812618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:32.323051236 +0000 UTC m=+995.254918747" watchObservedRunningTime="2026-11-03 20:13:32.325812618 +0000 UTC m=+995.257680126" Nov 03 20:13:32.357587 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:32.357541 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" podStartSLOduration=2.357526365 podStartE2EDuration="2.357526365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:32.35586357 +0000 UTC m=+995.287731076" watchObservedRunningTime="2026-11-03 20:13:32.357526365 +0000 UTC m=+995.289393876" Nov 03 20:13:32.384098 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:32.384066 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:32.385507 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:32.385478 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:13:32.385623 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:32.385527 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" podUID="75fc50e4-2ec7-4e3f-bf44-0572e998c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 20:13:32.390931 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:32.390902 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:32.392368 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:32.392336 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:13:32.392455 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:32.392391 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" podUID="eba28d4c-a907-4b6f-8d20-0a1baade4c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 20:13:32.577219 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:32.577170 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:32.578965 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:32.578932 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 20:13:32.579118 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:32.578995 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" podUID="f6f6c399-4702-41b4-b4f4-8920d34d7cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 20:13:33.396203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:33.395502 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:13:33.396203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:33.395560 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" podUID="75fc50e4-2ec7-4e3f-bf44-0572e998c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 20:13:33.396755 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:33.396207 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:13:33.396755 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:33.396243 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" podUID="eba28d4c-a907-4b6f-8d20-0a1baade4c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 20:13:33.582202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:33.581777 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 20:13:33.582202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:33.581859 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" podUID="f6f6c399-4702-41b4-b4f4-8920d34d7cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 20:13:34.398200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:34.396595 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:13:34.398200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:34.396666 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" podUID="75fc50e4-2ec7-4e3f-bf44-0572e998c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 20:13:34.410204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:34.402354 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:13:34.410204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:34.402412 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" podUID="eba28d4c-a907-4b6f-8d20-0a1baade4c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 20:13:34.582203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:34.581624 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 20:13:34.582203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:34.581689 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" podUID="f6f6c399-4702-41b4-b4f4-8920d34d7cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 20:13:34.865203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:34.864285 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:34.952204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:34.950440 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:34.964203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:34.956454 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:13:34.964203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:34.956514 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:34.989198 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:34.988699 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:35.065168 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:35.065108 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:35.091375 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:35.091321 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:35.165384 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:35.165278 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:35.390125 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:35.389664 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:13:35.390125 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:35.389729 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" podUID="75fc50e4-2ec7-4e3f-bf44-0572e998c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 20:13:35.394544 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:35.394514 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:13:35.394756 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:35.394730 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" podUID="eba28d4c-a907-4b6f-8d20-0a1baade4c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 20:13:35.670212 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:35.667708 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 20:13:35.670212 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:35.667771 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" podUID="f6f6c399-4702-41b4-b4f4-8920d34d7cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 20:13:35.896210 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:35.888787 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 20:13:35.896210 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:35.893322 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:13:35.974267 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:35.973395 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded" Nov 03 20:13:36.038206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:35.996684 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:36.038206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:36.022629 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 20:13:36.038206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:36.027320 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:13:36.038206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:36.027384 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:13:36.038206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:36.027424 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:13:36.097224 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:36.074219 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 20:13:36.198201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:36.197277 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:36.198201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:36.197747 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:13:36.198201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:36.197815 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:13:36.241261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:36.210328 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:36.241261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:36.210721 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:13:36.543486 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:36.543360 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:13:36.543791 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:36.543743 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" podUID="75fc50e4-2ec7-4e3f-bf44-0572e998c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 20:13:36.612322 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:36.596686 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 20:13:36.612322 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:36.596757 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" podUID="f6f6c399-4702-41b4-b4f4-8920d34d7cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 20:13:36.648664 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:36.648625 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:13:36.648934 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:36.648903 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" podUID="eba28d4c-a907-4b6f-8d20-0a1baade4c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 20:13:36.890625 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:36.890569 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:37.038609 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:37.038567 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:13:37.087228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:37.066427 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:37.089205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:37.088925 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:37.090229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:37.090201 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:37.223270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:37.219288 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:37.272560 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:37.238216 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:37.408087 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:37.408051 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:13:37.408367 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:37.408335 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" podUID="75fc50e4-2ec7-4e3f-bf44-0572e998c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 20:13:37.413628 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:37.412576 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:13:37.413628 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:37.413541 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" podUID="eba28d4c-a907-4b6f-8d20-0a1baade4c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 20:13:37.506577 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:37.506484 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:13:37.539225 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:37.539196 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:13:37.577644 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:37.574495 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:13:37.578781 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:37.578750 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 20:13:37.579013 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:37.578982 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" podUID="f6f6c399-4702-41b4-b4f4-8920d34d7cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 20:13:37.622132 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:37.622060 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:13:37.822667 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:37.822340 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:38.016666 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:38.016550 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 20:13:38.091300 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:38.088528 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 20:13:38.214590 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:38.214550 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 20:13:38.315249 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:38.315111 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:13:38.345390 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:38.345304 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:13:38.393833 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:38.393785 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:13:38.394129 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:38.394085 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" podUID="eba28d4c-a907-4b6f-8d20-0a1baade4c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 20:13:38.509048 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:38.509008 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:13:38.509281 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:38.509078 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" podUID="75fc50e4-2ec7-4e3f-bf44-0572e998c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 20:13:38.517388 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:38.517362 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:13:38.607893 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:38.607800 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 20:13:38.608445 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:38.608408 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" podUID="f6f6c399-4702-41b4-b4f4-8920d34d7cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 20:13:39.387204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:39.385599 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:13:39.387204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:39.385678 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" podUID="75fc50e4-2ec7-4e3f-bf44-0572e998c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 20:13:39.406218 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:39.397071 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:13:39.406218 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:39.397125 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" podUID="eba28d4c-a907-4b6f-8d20-0a1baade4c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 20:13:39.578199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:39.577560 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 20:13:39.578199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:39.577627 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" podUID="f6f6c399-4702-41b4-b4f4-8920d34d7cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 20:13:40.384524 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:40.384477 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:13:40.384710 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:40.384562 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" podUID="75fc50e4-2ec7-4e3f-bf44-0572e998c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 20:13:40.391235 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:40.391164 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:13:40.391611 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:40.391234 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" podUID="eba28d4c-a907-4b6f-8d20-0a1baade4c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 20:13:40.577802 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:40.577764 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 20:13:40.577980 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:40.577825 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" podUID="f6f6c399-4702-41b4-b4f4-8920d34d7cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 20:13:41.966999 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:41.966956 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:41.973200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:41.968556 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:42.020203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:42.011514 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:13:42.386306 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:42.386265 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:13:42.386495 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:42.386332 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" podUID="75fc50e4-2ec7-4e3f-bf44-0572e998c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:42.434202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:42.433292 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:13:42.434202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:42.433357 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" podUID="eba28d4c-a907-4b6f-8d20-0a1baade4c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:42.744256 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:42.742655 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:42.744256 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:42.744202 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:42.768702 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:42.765482 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:13:43.228807 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:43.228771 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:43.229359 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:43.229171 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:43.272210 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:43.272151 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:13:45.973203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:45.972323 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n"] Nov 03 20:13:45.979094 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:45.977446 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:45.991669 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:45.991635 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n"] Nov 03 20:13:46.061841 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.061805 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.062007 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.061941 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.062048 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.062019 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzjmc\" (UniqueName: \"kubernetes.io/projected/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-kube-api-access-dzjmc\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.062084 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.062071 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.062121 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.062101 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.062248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.062234 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.062296 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.062259 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.062369 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.062348 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.062414 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.062397 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.166200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.163514 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.166200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.163606 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.166200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.163720 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.166200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.163759 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzjmc\" (UniqueName: \"kubernetes.io/projected/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-kube-api-access-dzjmc\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.166200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.163812 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.166200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.163842 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.166200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.163928 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.166200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.163963 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.166200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.164053 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.166200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.164495 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.166200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.164739 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.166200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.164984 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.166200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.165208 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.166200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.165763 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.167025 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.166664 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.169257 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.169235 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.176551 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.176528 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.177660 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.177641 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzjmc\" (UniqueName: \"kubernetes.io/projected/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-kube-api-access-dzjmc\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.296388 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.296312 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:46.583262 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.583229 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n"] Nov 03 20:13:46.583518 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:13:46.583489 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477a6f6c_a2f6_4aa2_b9ed_f323e5bfc944.slice/crio-e25d55c19439b1fe5b57196d560aded4c4b167e57e7056567d7ef62fcb5eb7f9 WatchSource:0}: Error finding container e25d55c19439b1fe5b57196d560aded4c4b167e57e7056567d7ef62fcb5eb7f9: Status 404 returned error can't find the container with id e25d55c19439b1fe5b57196d560aded4c4b167e57e7056567d7ef62fcb5eb7f9 Nov 03 20:13:46.586558 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.586521 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:46.586649 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.586590 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:46.586649 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:46.586628 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:13:47.420373 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:47.420328 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" event={"ID":"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944","Type":"ContainerStarted","Data":"baad3552eded59fd592ea97eef9aa2baa2dedf2816289d8cbd996008e5585999"} Nov 03 20:13:47.421776 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:47.421749 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" event={"ID":"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944","Type":"ContainerStarted","Data":"e25d55c19439b1fe5b57196d560aded4c4b167e57e7056567d7ef62fcb5eb7f9"} Nov 03 20:13:47.459210 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:47.451146 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" podStartSLOduration=2.451124982 podStartE2EDuration="2.451124982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:47.450566459 +0000 UTC m=+1010.382433972" watchObservedRunningTime="2026-11-03 20:13:47.451124982 +0000 UTC m=+1010.382992495" Nov 03 20:13:48.297201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:48.297141 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:48.298755 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:48.298730 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:13:48.298875 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:48.298779 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" podUID="477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 20:13:49.296778 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:49.296737 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:13:49.297163 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:49.296810 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" podUID="477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 20:13:50.297630 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:50.297592 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:13:50.298008 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:50.297651 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" podUID="477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 20:13:51.296865 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:51.296825 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:13:51.297214 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:51.297164 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" podUID="477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 20:13:52.306208 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:52.299565 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:13:52.306208 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:52.299629 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" podUID="477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 20:13:53.300582 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:53.300546 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:13:53.300877 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:53.300846 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" podUID="477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 20:13:54.322961 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:54.322915 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:54.337559 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:54.337514 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:54.552334 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:54.552278 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:54.672278 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:54.672156 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:55.091726 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.091679 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:55.092373 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.091980 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:55.137206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.133281 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:55.301234 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.281587 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:55.362258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.341342 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:13:55.362258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.341421 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" podUID="477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:55.362258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.343576 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:55.362258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.347293 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:13:55.362258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.348153 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:13:55.362258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.348224 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" podUID="5120ad5e-81f7-43c1-9e1d-f0407339bb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:55.362258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.349221 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 20:13:55.362258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.349526 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:13:55.563659 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.563620 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:13:55.563937 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.563906 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" podUID="c1759364-4aca-4b1d-9f83-a81bc5829924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:55.576764 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.564245 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 20:13:55.576764 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.565484 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:13:55.672750 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.672638 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 20:13:55.712413 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.707561 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:13:55.750212 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.743666 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:55.750212 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.745719 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:55.750955 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.750931 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:13:55.760974 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.760888 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:13:55.832852 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.832822 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:13:55.941605 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.941524 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:13:55.962972 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:13:55.957121 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:13:58.208380 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:13:58.208340 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:14:16.010543 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.010466 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq"] Nov 03 20:14:16.014468 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.014450 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.031055 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.031015 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.036427 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.036387 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.036839 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.036815 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.040409 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.040352 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.048258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.040440 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.048258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.041897 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.048258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.042737 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bkzt\" (UniqueName: \"kubernetes.io/projected/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-kube-api-access-6bkzt\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.048258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.042860 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.048258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.042931 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.060806 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.058929 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq"] Nov 03 20:14:16.144209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.144139 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.144404 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.144229 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.144404 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.144249 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.144404 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.144295 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.144404 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.144339 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.144404 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.144362 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.144404 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.144399 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.144756 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.144467 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.144756 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.144553 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bkzt\" (UniqueName: \"kubernetes.io/projected/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-kube-api-access-6bkzt\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.144756 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.144609 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.144756 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.144620 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.144756 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.144689 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.144955 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.144837 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.145248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.145230 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.147080 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.147059 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.148449 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.148415 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.156597 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.156569 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.156788 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.156763 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bkzt\" (UniqueName: \"kubernetes.io/projected/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-kube-api-access-6bkzt\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.337872 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.337839 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:16.624071 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.624040 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq"] Nov 03 20:14:16.624228 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:14:16.624161 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e6ea59_b2c6_409d_b2e8_4a4eec65938d.slice/crio-b75caeefb166dc571500b68351b7d3a70c8e1a102733027fdbf43a5d6d0cd785 WatchSource:0}: Error finding container b75caeefb166dc571500b68351b7d3a70c8e1a102733027fdbf43a5d6d0cd785: Status 404 returned error can't find the container with id b75caeefb166dc571500b68351b7d3a70c8e1a102733027fdbf43a5d6d0cd785 Nov 03 20:14:16.626789 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.626756 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:16.626919 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.626817 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:16.626919 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:16.626856 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:17.100559 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:17.100520 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" event={"ID":"11e6ea59-b2c6-409d-b2e8-4a4eec65938d","Type":"ContainerStarted","Data":"186a93d3ce6d92dcbcf17fd9fbb97331e9819264f37af51deeaca7b5f61bbb47"} Nov 03 20:14:17.101032 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:17.100614 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" event={"ID":"11e6ea59-b2c6-409d-b2e8-4a4eec65938d","Type":"ContainerStarted","Data":"b75caeefb166dc571500b68351b7d3a70c8e1a102733027fdbf43a5d6d0cd785"} Nov 03 20:14:17.128926 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:17.128858 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" podStartSLOduration=2.128838936 podStartE2EDuration="2.128838936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:17.125489076 +0000 UTC m=+1040.057356610" watchObservedRunningTime="2026-11-03 20:14:17.128838936 +0000 UTC m=+1040.060706447" Nov 03 20:14:17.339955 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:17.339920 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:17.344313 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:17.344282 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:14:17.344569 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:17.344535 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" podUID="11e6ea59-b2c6-409d-b2e8-4a4eec65938d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 20:14:18.344202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:18.342546 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:14:18.344202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:18.342623 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" podUID="11e6ea59-b2c6-409d-b2e8-4a4eec65938d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 20:14:19.362200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:19.361565 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:14:19.362200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:19.361639 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" podUID="11e6ea59-b2c6-409d-b2e8-4a4eec65938d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 20:14:19.508331 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:19.508271 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:19.766523 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:19.766390 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:19.836310 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:19.836263 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:19.955202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:19.952264 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:19.969229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:19.969171 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.047567 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:20.047455 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.109200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:20.106312 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.271239 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:20.247269 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.348747 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:20.348710 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:14:20.349064 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:20.349017 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" podUID="11e6ea59-b2c6-409d-b2e8-4a4eec65938d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 20:14:20.781261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:20.777239 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.781261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:20.777994 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:14:20.838000 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:20.837955 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.856220 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:20.846288 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:14:21.004201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:21.001476 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 20:14:21.004201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:21.003405 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:14:21.040259 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:21.036327 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:21.056215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:21.055912 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:14:21.067138 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:21.067093 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:21.117633 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:21.117256 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:21.117633 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:21.117537 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:14:21.117633 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:21.117599 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:14:21.119338 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:21.119309 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Nov 03 20:14:21.468616 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:21.468547 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:21.567439 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:21.567381 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:14:21.630326 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:21.629310 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" podUID="11e6ea59-b2c6-409d-b2e8-4a4eec65938d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 20:14:21.734459 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:21.734361 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:14:21.899383 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:21.857741 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:22.114256 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:22.022461 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:14:22.114686 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:22.042303 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:22.114876 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:22.106740 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" podUID="34a102a4-f391-4d65-8a93-324a300363cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": context deadline exceeded" Nov 03 20:14:22.114985 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:22.113421 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 20:14:22.245202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:22.243460 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:22.475287 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:22.475192 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:14:22.475577 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:22.475522 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" podUID="11e6ea59-b2c6-409d-b2e8-4a4eec65938d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 20:14:22.507975 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:22.507932 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:22.513153 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:22.513110 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:22.526208 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:22.523287 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:14:22.661309 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:22.661271 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:14:22.702215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:22.694554 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:14:22.702393 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:22.702316 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:14:22.702393 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:22.702389 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:14:22.750251 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:22.746490 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:14:23.339200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:23.338592 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:14:23.339200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:23.338656 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" podUID="11e6ea59-b2c6-409d-b2e8-4a4eec65938d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 20:14:24.343658 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:24.343619 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:14:24.344521 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:24.343684 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" podUID="11e6ea59-b2c6-409d-b2e8-4a4eec65938d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 20:14:25.338301 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.338265 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:14:25.338471 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.338318 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" podUID="11e6ea59-b2c6-409d-b2e8-4a4eec65938d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 20:14:25.817353 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.817314 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds"] Nov 03 20:14:25.823741 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.823716 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:25.840635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.840593 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds"] Nov 03 20:14:25.857880 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.857847 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm"] Nov 03 20:14:25.863894 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.863863 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:25.880346 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.880321 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm"] Nov 03 20:14:25.974325 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.974283 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:25.974564 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.974349 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:25.974564 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.974411 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:25.974564 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.974514 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:25.974564 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.974557 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brvkl\" (UniqueName: \"kubernetes.io/projected/da35ea39-491f-4d1b-b762-3d231a0aa1f9-kube-api-access-brvkl\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:25.974929 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.974652 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kgwq\" (UniqueName: \"kubernetes.io/projected/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-kube-api-access-9kgwq\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:25.974929 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.974694 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:25.974929 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.974849 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:25.974929 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.974884 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:25.975205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.974974 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:25.975205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.975015 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:25.975205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.975082 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:25.975205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.975126 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:25.975457 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.975224 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:25.975457 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.975310 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:25.975457 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.975385 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:25.975645 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.975456 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:25.975645 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:25.975557 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:26.077503 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.077413 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:26.077503 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.077476 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:26.077737 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.077514 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:26.077737 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.077557 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:26.077737 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.077602 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:26.077737 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.077647 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:26.077737 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.077703 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:26.077737 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.077733 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:26.078051 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.077792 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:26.078051 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.077871 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:26.078051 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.077911 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:26.078051 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.077950 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:26.078051 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.077957 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:26.078051 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.078020 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:26.078438 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.078064 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:26.078438 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.078087 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:26.078438 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.078102 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:26.078438 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.078124 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:26.078438 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.078218 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvkl\" (UniqueName: \"kubernetes.io/projected/da35ea39-491f-4d1b-b762-3d231a0aa1f9-kube-api-access-brvkl\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:26.078438 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.078283 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kgwq\" (UniqueName: \"kubernetes.io/projected/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-kube-api-access-9kgwq\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:26.078438 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.078319 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:26.078438 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.078407 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:26.079522 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.078911 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:26.079522 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.078970 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:26.079522 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.079313 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:26.079522 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.079399 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:26.079522 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.079474 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:26.080369 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.080345 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:26.081155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.081130 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:26.081400 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.081376 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:26.081642 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.081621 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:26.082903 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.082878 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:26.091216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.091139 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:26.091599 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.091579 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brvkl\" (UniqueName: \"kubernetes.io/projected/da35ea39-491f-4d1b-b762-3d231a0aa1f9-kube-api-access-brvkl\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:26.091668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.091589 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kgwq\" (UniqueName: \"kubernetes.io/projected/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-kube-api-access-9kgwq\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:26.092107 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.092084 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:26.167780 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.167745 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:26.187672 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.187639 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:26.350109 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.349946 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:26.351949 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.351683 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:26.353008 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.352918 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:14:26.467880 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.467846 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds"] Nov 03 20:14:26.469292 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:14:26.469255 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda35ea39_491f_4d1b_b762_3d231a0aa1f9.slice/crio-0aded6041af58096a6cb0e8138b5fc357e1c0bf55f2745547a7ced1bce82290d WatchSource:0}: Error finding container 0aded6041af58096a6cb0e8138b5fc357e1c0bf55f2745547a7ced1bce82290d: Status 404 returned error can't find the container with id 0aded6041af58096a6cb0e8138b5fc357e1c0bf55f2745547a7ced1bce82290d Nov 03 20:14:26.472699 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.472662 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:26.472785 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.472735 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:26.472785 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.472764 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:26.501308 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.501281 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm"] Nov 03 20:14:26.505456 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:14:26.505428 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dad3fad_396e_4647_a9bf_a7a906fdd1fe.slice/crio-c34953b9bc5f8207ffe05156a14dffb9cec55e989cf6dc2354aa5894c09d0128 WatchSource:0}: Error finding container c34953b9bc5f8207ffe05156a14dffb9cec55e989cf6dc2354aa5894c09d0128: Status 404 returned error can't find the container with id c34953b9bc5f8207ffe05156a14dffb9cec55e989cf6dc2354aa5894c09d0128 Nov 03 20:14:26.507408 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.507371 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:26.507508 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.507454 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:26.507508 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.507500 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:26.572501 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.572412 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" event={"ID":"6dad3fad-396e-4647-a9bf-a7a906fdd1fe","Type":"ContainerStarted","Data":"c34953b9bc5f8207ffe05156a14dffb9cec55e989cf6dc2354aa5894c09d0128"} Nov 03 20:14:26.576303 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.576266 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" event={"ID":"da35ea39-491f-4d1b-b762-3d231a0aa1f9","Type":"ContainerStarted","Data":"469b703fc5a46fca5dd115ead8e42c7bf7be88d4785e1abe8615e63a6948ace8"} Nov 03 20:14:26.576618 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.576324 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" event={"ID":"da35ea39-491f-4d1b-b762-3d231a0aa1f9","Type":"ContainerStarted","Data":"0aded6041af58096a6cb0e8138b5fc357e1c0bf55f2745547a7ced1bce82290d"} Nov 03 20:14:26.601133 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:26.601036 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" podStartSLOduration=1.601016858 podStartE2EDuration="1.601016858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:26.598877385 +0000 UTC m=+1049.530744890" watchObservedRunningTime="2026-11-03 20:14:26.601016858 +0000 UTC m=+1049.532884371" Nov 03 20:14:27.168110 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:27.168071 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:27.169509 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:27.169482 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:14:27.169635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:27.169533 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" podUID="da35ea39-491f-4d1b-b762-3d231a0aa1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 20:14:27.584144 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:27.584100 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" event={"ID":"6dad3fad-396e-4647-a9bf-a7a906fdd1fe","Type":"ContainerStarted","Data":"ab1aca5343b895af833d4d40863d1d20d4d364a3edae815679eac4e90e777704"} Nov 03 20:14:27.613477 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:27.613431 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" podStartSLOduration=2.613417895 podStartE2EDuration="2.613417895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:27.610145234 +0000 UTC m=+1050.542012747" watchObservedRunningTime="2026-11-03 20:14:27.613417895 +0000 UTC m=+1050.545285406" Nov 03 20:14:28.169196 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:28.168823 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:14:28.169196 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:28.168887 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" podUID="da35ea39-491f-4d1b-b762-3d231a0aa1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 20:14:28.188334 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:28.188303 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:28.189821 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:28.189802 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:14:28.189884 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:28.189846 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" podUID="6dad3fad-396e-4647-a9bf-a7a906fdd1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 20:14:28.295218 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:14:28.294970 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:14:29.182203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:29.181475 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:14:29.182203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:29.181536 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" podUID="da35ea39-491f-4d1b-b762-3d231a0aa1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 20:14:29.188411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:29.188318 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:14:29.188411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:29.188362 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" podUID="6dad3fad-396e-4647-a9bf-a7a906fdd1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 20:14:30.182094 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:30.182047 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:14:30.182336 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:30.182117 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" podUID="da35ea39-491f-4d1b-b762-3d231a0aa1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 20:14:30.188325 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:30.188295 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:14:30.188433 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:30.188361 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" podUID="6dad3fad-396e-4647-a9bf-a7a906fdd1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 20:14:31.174437 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.174400 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:14:31.174753 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.174704 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" podUID="da35ea39-491f-4d1b-b762-3d231a0aa1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 20:14:31.175351 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.175313 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj"] Nov 03 20:14:31.181433 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.181411 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.188775 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.188748 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:14:31.189965 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.189939 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" podUID="6dad3fad-396e-4647-a9bf-a7a906fdd1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 20:14:31.190383 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.190077 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj"] Nov 03 20:14:31.275248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.274592 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b614fc-7245-4556-886c-3f9118db35ad-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.275248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.274680 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.275248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.274749 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.275248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.274888 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.275248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.274950 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b614fc-7245-4556-886c-3f9118db35ad-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.275248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.275024 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwzqt\" (UniqueName: \"kubernetes.io/projected/46b614fc-7245-4556-886c-3f9118db35ad-kube-api-access-qwzqt\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.275248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.275078 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b614fc-7245-4556-886c-3f9118db35ad-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.275248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.275136 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.275248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.275240 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.288594 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.288556 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk"] Nov 03 20:14:31.293625 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.293579 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.312767 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.312735 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk"] Nov 03 20:14:31.383199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.382590 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.383199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.382666 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.383199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.382715 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.383199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.382762 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.383199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.382803 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.383199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.382830 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b614fc-7245-4556-886c-3f9118db35ad-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.383199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.382892 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.383199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.382959 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwzqt\" (UniqueName: \"kubernetes.io/projected/46b614fc-7245-4556-886c-3f9118db35ad-kube-api-access-qwzqt\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.383199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.383020 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b614fc-7245-4556-886c-3f9118db35ad-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.383199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.383081 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.383199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.383143 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.383199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.383190 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.383199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.383217 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmnbm\" (UniqueName: \"kubernetes.io/projected/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-kube-api-access-vmnbm\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.383993 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.383280 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.383993 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.383342 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b614fc-7245-4556-886c-3f9118db35ad-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.383993 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.383381 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.383993 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.383460 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.383993 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.383511 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.384265 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.384080 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.388202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.387595 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.388202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.388133 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.388428 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.388229 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b614fc-7245-4556-886c-3f9118db35ad-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.388428 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.388395 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.405204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.392829 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b614fc-7245-4556-886c-3f9118db35ad-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.405204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.401295 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp"] Nov 03 20:14:31.439246 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.407451 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.439246 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.408574 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b614fc-7245-4556-886c-3f9118db35ad-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.439246 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.420728 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr"] Nov 03 20:14:31.452892 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.440201 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.476532 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwzqt\" (UniqueName: \"kubernetes.io/projected/46b614fc-7245-4556-886c-3f9118db35ad-kube-api-access-qwzqt\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.489763 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.490317 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.490358 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.490410 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.490443 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.490559 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.490583 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.490625 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.490653 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.490678 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmnbm\" (UniqueName: \"kubernetes.io/projected/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-kube-api-access-vmnbm\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.490854 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.490888 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qngml\" (UniqueName: \"kubernetes.io/projected/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-kube-api-access-qngml\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.490920 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.490989 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.491015 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.491060 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.491108 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.491159 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.491577 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.490127 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.492215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.492146 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.494900 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.492911 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.494900 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.493737 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.494900 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.494443 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.502381 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.502353 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp"] Nov 03 20:14:31.505610 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.505552 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:31.507340 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.507313 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr"] Nov 03 20:14:31.589513 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.587190 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmnbm\" (UniqueName: \"kubernetes.io/projected/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-kube-api-access-vmnbm\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.589513 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.587942 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.589513 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.588566 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.589513 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.589002 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.596544 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.596513 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.596673 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.596575 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.596727 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.596698 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dqv6\" (UniqueName: \"kubernetes.io/projected/370a7106-3c26-451f-8ad2-76a27a576f64-kube-api-access-5dqv6\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.596801 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.596784 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qngml\" (UniqueName: \"kubernetes.io/projected/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-kube-api-access-qngml\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.596845 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.596821 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.596901 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.596886 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370a7106-3c26-451f-8ad2-76a27a576f64-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.596938 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.596918 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.596998 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.596985 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.597046 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.597033 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.597093 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.597080 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.597127 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.597113 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.597169 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.597139 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.597539 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.597478 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.597977 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.597908 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.597977 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.597937 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.598144 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.598034 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370a7106-3c26-451f-8ad2-76a27a576f64-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.598144 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.598108 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.598144 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.598126 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370a7106-3c26-451f-8ad2-76a27a576f64-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.598352 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.598155 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.598352 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.598189 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.598914 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.598867 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.599374 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.599353 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.599631 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.599571 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.603540 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.603517 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.603872 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.603848 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.607879 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.607442 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:31.609556 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.609533 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.618154 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.618132 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qngml\" (UniqueName: \"kubernetes.io/projected/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-kube-api-access-qngml\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.699676 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.699647 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370a7106-3c26-451f-8ad2-76a27a576f64-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.699796 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.699745 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.699796 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.699774 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370a7106-3c26-451f-8ad2-76a27a576f64-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.699959 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.699936 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.700046 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.699994 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.700112 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.700051 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dqv6\" (UniqueName: \"kubernetes.io/projected/370a7106-3c26-451f-8ad2-76a27a576f64-kube-api-access-5dqv6\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.700201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.700135 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370a7106-3c26-451f-8ad2-76a27a576f64-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.700311 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.700290 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.700381 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.700353 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.701781 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.700915 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.701781 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.701020 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.701781 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.701092 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.701781 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.701729 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.702158 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.702134 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370a7106-3c26-451f-8ad2-76a27a576f64-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.711997 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.704907 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370a7106-3c26-451f-8ad2-76a27a576f64-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.780080 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.780009 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370a7106-3c26-451f-8ad2-76a27a576f64-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.783815 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.783745 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dqv6\" (UniqueName: \"kubernetes.io/projected/370a7106-3c26-451f-8ad2-76a27a576f64-kube-api-access-5dqv6\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.793704 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.793645 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:31.915845 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.915580 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:31.921209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:31.921092 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:32.008080 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.004565 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:32.008080 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.004652 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:32.008080 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.004692 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:32.008080 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.007501 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj"] Nov 03 20:14:32.179640 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.179163 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:14:32.179640 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.179243 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" podUID="da35ea39-491f-4d1b-b762-3d231a0aa1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 20:14:32.193255 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.193223 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:14:32.193710 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.193317 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" podUID="6dad3fad-396e-4647-a9bf-a7a906fdd1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 20:14:32.301883 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.288204 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk"] Nov 03 20:14:32.301883 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.296258 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:32.301883 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.296343 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:32.301883 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.296391 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:32.492273 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.484275 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp"] Nov 03 20:14:32.506473 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.506443 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr"] Nov 03 20:14:32.515115 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.514651 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:32.515115 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.514737 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:32.515115 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.514778 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:32.515822 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.515789 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:32.515918 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.515863 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:32.515974 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.515926 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:14:32.679525 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.679425 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" event={"ID":"370a7106-3c26-451f-8ad2-76a27a576f64","Type":"ContainerStarted","Data":"066ad8bc1eb5d89cc84f0131751f933d3cb4b816ea2cf239d8278f7c6c6da114"} Nov 03 20:14:32.690935 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.690842 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" event={"ID":"0563a3f2-35d7-4d0a-a99d-b4d1e9277855","Type":"ContainerStarted","Data":"facd2aa75b70247e8a773b85b70f5e1273253e9170c8fefcf1bd3358af89824a"} Nov 03 20:14:32.692574 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.692501 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" event={"ID":"ea246681-cf7a-40b9-97d1-9c1d2594c3ec","Type":"ContainerStarted","Data":"3c345e9f9a80b5bd5b42eec4ac418ec33135ebc035ccf9774d865b4e14e4f103"} Nov 03 20:14:32.696943 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.695796 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" event={"ID":"46b614fc-7245-4556-886c-3f9118db35ad","Type":"ContainerStarted","Data":"7c99c1a48b75bbf71559dc597000e0d21065c8b2af0571db86207e4f50380bcf"} Nov 03 20:14:32.696943 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.695844 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" event={"ID":"46b614fc-7245-4556-886c-3f9118db35ad","Type":"ContainerStarted","Data":"6660614056afc390353270d1ab3eb20ba817a4371753a46e5e56d2397671f64f"} Nov 03 20:14:32.762226 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:32.756325 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" podStartSLOduration=1.7562922250000002 podStartE2EDuration="1.756292225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:32.755333853 +0000 UTC m=+1055.687201365" watchObservedRunningTime="2026-11-03 20:14:32.756292225 +0000 UTC m=+1055.688159737" Nov 03 20:14:33.182846 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:33.182797 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:14:33.183150 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:33.183119 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" podUID="da35ea39-491f-4d1b-b762-3d231a0aa1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 20:14:33.201062 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:33.188836 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:14:33.201062 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:33.188889 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" podUID="6dad3fad-396e-4647-a9bf-a7a906fdd1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 20:14:33.509160 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:33.509077 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:33.509504 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:33.509478 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:14:33.509670 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:33.509641 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" podUID="46b614fc-7245-4556-886c-3f9118db35ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:14:33.874060 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:33.874020 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" event={"ID":"0563a3f2-35d7-4d0a-a99d-b4d1e9277855","Type":"ContainerStarted","Data":"7bbd1b1dcbce6c44e04b2b4f72e842a40bd47ce1d3ef8c330737d3ab5b2dfc74"} Nov 03 20:14:33.909325 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:33.909283 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" event={"ID":"ea246681-cf7a-40b9-97d1-9c1d2594c3ec","Type":"ContainerStarted","Data":"5ee2fb2cb5bab41990dacff40ca3ad55ce1d6623450c2264e1a870d6db532581"} Nov 03 20:14:33.946429 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:33.929600 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" event={"ID":"370a7106-3c26-451f-8ad2-76a27a576f64","Type":"ContainerStarted","Data":"c7a585b18ff8f3f2f738b5e81788d39d50a8f4536a42334b444a01bd333a7165"} Nov 03 20:14:33.946429 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:33.929683 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:33.946429 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:33.936121 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" podStartSLOduration=2.9361005970000003 podStartE2EDuration="2.936100597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:33.933399115 +0000 UTC m=+1056.865266632" watchObservedRunningTime="2026-11-03 20:14:33.936100597 +0000 UTC m=+1056.867968109" Nov 03 20:14:33.971976 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:33.947230 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:33.974024 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:33.973984 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:14:33.974151 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:33.974047 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" podUID="370a7106-3c26-451f-8ad2-76a27a576f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:14:34.019201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:34.014862 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:14:34.019201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:34.014922 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" podUID="0563a3f2-35d7-4d0a-a99d-b4d1e9277855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 20:14:34.041201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:34.038808 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" podStartSLOduration=3.03878789 podStartE2EDuration="3.03878789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:33.988869763 +0000 UTC m=+1056.920737275" watchObservedRunningTime="2026-11-03 20:14:34.03878789 +0000 UTC m=+1056.970655405" Nov 03 20:14:34.101234 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:34.099747 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" podStartSLOduration=3.099722294 podStartE2EDuration="3.099722294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:34.053226552 +0000 UTC m=+1056.985094064" watchObservedRunningTime="2026-11-03 20:14:34.099722294 +0000 UTC m=+1057.031589806" Nov 03 20:14:34.170449 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:34.170353 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:14:34.170727 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:34.170695 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" podUID="da35ea39-491f-4d1b-b762-3d231a0aa1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 20:14:34.189672 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:34.189635 4918 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:14:34.189940 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:34.189899 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" podUID="6dad3fad-396e-4647-a9bf-a7a906fdd1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 20:14:34.509226 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:34.506573 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:14:34.509226 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:34.506639 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" podUID="46b614fc-7245-4556-886c-3f9118db35ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:14:34.611288 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:34.611245 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:34.611685 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:34.611661 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:14:34.611851 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:34.611829 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" podUID="ea246681-cf7a-40b9-97d1-9c1d2594c3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 20:14:34.681200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:34.680501 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:34.918623 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:34.918583 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:14:34.918925 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:34.918894 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" podUID="0563a3f2-35d7-4d0a-a99d-b4d1e9277855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 20:14:34.923534 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:34.923504 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:14:34.923744 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:34.923717 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" podUID="370a7106-3c26-451f-8ad2-76a27a576f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:14:34.985926 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:34.985881 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 20:14:35.264261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:35.249276 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded" Nov 03 20:14:35.264261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:35.249703 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:35.412511 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:35.412477 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:35.452271 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:35.439820 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:35.530290 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:35.507838 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:14:35.530290 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:35.507896 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" podUID="46b614fc-7245-4556-886c-3f9118db35ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:14:35.587387 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:35.587337 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:35.588518 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:35.588322 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:14:35.620573 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:35.618609 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:35.620573 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:35.619012 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:14:35.620573 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:35.619060 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" podUID="ea246681-cf7a-40b9-97d1-9c1d2594c3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 20:14:35.967204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:35.931900 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:14:36.364223 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:35.973259 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:36.364223 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:35.974886 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:14:36.392619 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:36.045200 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:14:36.392920 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:36.392891 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" podUID="0563a3f2-35d7-4d0a-a99d-b4d1e9277855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 20:14:36.393083 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:36.289271 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:36.393914 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:36.393893 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:14:36.690602 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:36.690501 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:14:36.746828 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:36.068037 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 20:14:36.834234 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:36.289555 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded" Nov 03 20:14:36.901361 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:36.093430 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:14:36.901361 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:36.893330 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" podUID="370a7106-3c26-451f-8ad2-76a27a576f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:14:36.901361 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:36.602633 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:14:36.901361 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:36.893423 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" podUID="46b614fc-7245-4556-886c-3f9118db35ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:14:36.901361 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:36.830518 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:14:36.901361 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:36.893479 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" podUID="ea246681-cf7a-40b9-97d1-9c1d2594c3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 20:14:36.955301 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:36.938275 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:14:37.062238 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:37.050259 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:14:37.109690 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:37.109643 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:37.172199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:37.171532 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" podUID="c9493241-95b1-4b6d-9290-c677b2a55000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded" Nov 03 20:14:37.205203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:37.173023 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:14:37.205203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:37.173078 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" podUID="0563a3f2-35d7-4d0a-a99d-b4d1e9277855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 20:14:37.410231 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:37.408771 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 20:14:37.410231 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:37.409213 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:14:37.417263 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:37.412065 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 20:14:37.504205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:37.477393 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:14:37.739731 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:37.739636 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" podUID="370a7106-3c26-451f-8ad2-76a27a576f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:14:37.940201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:37.549432 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:14:38.089732 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.089693 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" podUID="46b614fc-7245-4556-886c-3f9118db35ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:14:38.113227 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:37.914302 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 20:14:38.142366 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:37.914321 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded" Nov 03 20:14:38.170300 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:37.957636 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:38.170915 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.078340 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:14:38.178222 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.173810 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7qnhr" podUID="34a102a4-f391-4d65-8a93-324a300363cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:38.197433 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:37.914264 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded" Nov 03 20:14:38.197733 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:37.776357 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:14:38.197905 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.197869 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" podUID="ea246681-cf7a-40b9-97d1-9c1d2594c3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 20:14:38.198032 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.123706 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:38.219354 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:37.957564 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:14:38.219694 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.219660 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" podUID="370a7106-3c26-451f-8ad2-76a27a576f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:14:38.219926 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.219912 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:14:38.234817 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.083259 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-bzdwr" podUID="fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:38.374262 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.346167 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:14:38.604540 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.604487 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" podUID="0563a3f2-35d7-4d0a-a99d-b4d1e9277855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 20:14:38.608327 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.419035 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:14:38.637307 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.419056 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 20:14:38.654454 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.418980 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:38.666083 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.513594 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:14:38.666366 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.666318 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" podUID="46b614fc-7245-4556-886c-3f9118db35ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:14:38.666731 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.628039 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:14:38.717345 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.670243 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" podUID="ea246681-cf7a-40b9-97d1-9c1d2594c3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 20:14:39.003253 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.991821 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:14:39.003253 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.991881 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" podUID="370a7106-3c26-451f-8ad2-76a27a576f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:14:39.003253 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.991972 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:14:39.003253 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:38.991997 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" podUID="0563a3f2-35d7-4d0a-a99d-b4d1e9277855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 20:14:39.157142 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:39.157095 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:39.182584 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:39.171006 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 20:14:39.208410 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:39.208359 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:39.227200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:39.192956 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 20:14:39.244237 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:39.232110 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:14:39.540204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:39.516053 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:39.557356 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:39.557314 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:14:39.557645 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:39.557613 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" podUID="46b614fc-7245-4556-886c-3f9118db35ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:14:39.620708 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:39.620663 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:39.648378 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:39.648338 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:14:39.648671 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:39.648638 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" podUID="ea246681-cf7a-40b9-97d1-9c1d2594c3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 20:14:39.686205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:39.678984 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 20:14:39.691535 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:39.681974 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 20:14:39.926205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:39.917561 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:14:39.926205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:39.917634 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" podUID="0563a3f2-35d7-4d0a-a99d-b4d1e9277855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 20:14:39.951200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:39.937084 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:14:39.954300 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:39.954268 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:14:39.954548 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:39.954504 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" podUID="370a7106-3c26-451f-8ad2-76a27a576f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:14:39.981742 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:39.981577 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:14:40.297257 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:40.293509 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 20:14:40.508205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:40.503932 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 20:14:40.656201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:40.623876 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:14:40.680203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:40.656508 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" podUID="ea246681-cf7a-40b9-97d1-9c1d2594c3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 20:14:40.688347 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:40.665035 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:14:40.694469 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:40.694435 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:14:40.694720 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:40.694690 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" podUID="46b614fc-7245-4556-886c-3f9118db35ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:14:40.761231 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:40.759324 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:40.767104 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:40.767051 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:41.030314 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:40.997477 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:14:41.089363 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:41.089313 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" podUID="370a7106-3c26-451f-8ad2-76a27a576f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:14:41.091727 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:41.037095 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:14:41.092104 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:41.092073 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" podUID="0563a3f2-35d7-4d0a-a99d-b4d1e9277855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 20:14:41.177339 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:41.177303 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:14:41.269211 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:41.261479 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:14:41.307260 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:41.301045 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:14:41.362211 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:41.335575 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:14:41.507353 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:41.507306 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:14:41.507553 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:41.507373 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" podUID="46b614fc-7245-4556-886c-3f9118db35ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:14:41.609652 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:41.609614 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:14:41.609855 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:41.609699 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" podUID="ea246681-cf7a-40b9-97d1-9c1d2594c3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 20:14:41.917335 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:41.917241 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:14:41.917335 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:41.917313 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" podUID="0563a3f2-35d7-4d0a-a99d-b4d1e9277855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 20:14:41.921461 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:41.921427 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:14:41.921610 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:41.921478 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" podUID="370a7106-3c26-451f-8ad2-76a27a576f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:14:42.823936 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:42.823868 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:42.825971 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:42.825946 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:42.827462 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:42.827428 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:14:42.930587 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:42.930525 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:42.932524 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:42.932503 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:42.933525 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:42.933507 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:14:42.933903 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:42.933886 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:42.935255 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:42.935239 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:42.936835 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:42.936815 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:14:43.079795 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:43.079696 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:43.080883 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:43.080854 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:43.081596 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:43.081572 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:14:53.772494 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:53.772434 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:54.013230 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:53.999445 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Nov 03 20:14:54.164703 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:54.164658 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 20:14:54.197800 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:54.197354 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:54.222202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:54.218258 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:54.308516 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:54.308461 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 20:14:54.559202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:54.541325 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:54.756383 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:54.756321 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:54.785358 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:54.785258 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 20:14:54.786617 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:54.786592 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:14:54.993934 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:54.991572 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.211201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:55.205171 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.211201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:55.205989 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:14:55.211201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:55.207418 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.211201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:55.207799 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:14:55.232911 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:55.232855 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.378328 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:55.378260 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 20:14:55.435565 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:55.435379 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:14:55.435565 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:55.435455 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:14:55.531042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:55.530948 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:14:55.531314 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:55.531281 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" podUID="c1759364-4aca-4b1d-9f83-a81bc5829924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.543413 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:55.543354 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:14:55.543949 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:55.543927 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:14:55.782253 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:55.781144 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.782844 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:55.782811 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:14:55.809741 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:55.809689 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:56.216360 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:56.216304 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:56.222581 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:56.222532 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 20:14:56.280199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:56.258902 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:56.468250 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:56.461447 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 20:14:56.584641 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:56.584320 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:56.821246 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:56.814660 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:56.839373 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:56.839315 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:56.923541 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:56.923499 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:14:56.923827 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:56.923796 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" podUID="477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:57.224259 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:57.220764 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded" Nov 03 20:14:57.410209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:57.225028 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:57.508708 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:57.508600 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 20:14:57.636719 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:57.636676 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:14:57.657549 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:57.653469 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 20:14:57.738353 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:57.738280 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:57.902208 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:57.897764 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:57.965038 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:57.965005 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:14:58.178748 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:58.178505 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:58.218893 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:58.218862 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:14:58.230853 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:58.230818 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:14:58.231915 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:58.231895 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:14:58.260883 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:58.260850 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:14:58.273197 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:58.267076 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:14:58.273197 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:14:58.267912 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:14:58.321844 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:14:58.321814 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:15:02.946443 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:02.946400 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9"] Nov 03 20:15:02.952883 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:02.952855 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9" Nov 03 20:15:02.957907 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:02.957879 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:15:02.958049 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:02.957996 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:02.961946 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:02.961919 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9"] Nov 03 20:15:03.018080 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:03.018049 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc99981c-kqrz9\" (UID: \"28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9" Nov 03 20:15:03.018279 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:03.018091 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdjl4\" (UniqueName: \"kubernetes.io/projected/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-kube-api-access-cdjl4\") pod \"maas-api-revoke-sub-bc99981c-kqrz9\" (UID: \"28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9" Nov 03 20:15:03.018279 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:03.018199 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc99981c-kqrz9\" (UID: \"28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9" Nov 03 20:15:03.119658 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:03.119611 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc99981c-kqrz9\" (UID: \"28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9" Nov 03 20:15:03.119658 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:03.119660 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdjl4\" (UniqueName: \"kubernetes.io/projected/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-kube-api-access-cdjl4\") pod \"maas-api-revoke-sub-bc99981c-kqrz9\" (UID: \"28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9" Nov 03 20:15:03.119919 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:03.119707 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc99981c-kqrz9\" (UID: \"28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9" Nov 03 20:15:03.120498 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:03.120471 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc99981c-kqrz9\" (UID: \"28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9" Nov 03 20:15:03.122106 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:03.122079 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc99981c-kqrz9\" (UID: \"28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9" Nov 03 20:15:03.134371 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:03.134342 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdjl4\" (UniqueName: \"kubernetes.io/projected/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-kube-api-access-cdjl4\") pod \"maas-api-revoke-sub-bc99981c-kqrz9\" (UID: \"28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9" Nov 03 20:15:03.274043 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:03.273954 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9" Nov 03 20:15:03.575053 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:15:03.574996 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a0d3ab_cf16_4f2c_bb86_fb2e8bfb28ce.slice/crio-ac6c01df1b9265e1ff5ec85b8a322db41866102c9f8eaf125a3f9e383d46ccab WatchSource:0}: Error finding container ac6c01df1b9265e1ff5ec85b8a322db41866102c9f8eaf125a3f9e383d46ccab: Status 404 returned error can't find the container with id ac6c01df1b9265e1ff5ec85b8a322db41866102c9f8eaf125a3f9e383d46ccab Nov 03 20:15:03.577427 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:03.577393 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9"] Nov 03 20:15:03.580225 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:03.580129 4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:15:04.594005 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:04.593964 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9" event={"ID":"28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce","Type":"ContainerStarted","Data":"ac6c01df1b9265e1ff5ec85b8a322db41866102c9f8eaf125a3f9e383d46ccab"} Nov 03 20:15:06.614564 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:06.614523 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9" event={"ID":"28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce","Type":"ContainerStarted","Data":"c85f2497a8d10944c107d56f964f9183781adbc1e2919fad93ff55420eb85f3c"} Nov 03 20:15:06.641543 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:06.641481 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9" podStartSLOduration=2.050858634 podStartE2EDuration="4.641462773s" totalImagesPullingTime="2.590604139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:06.638108139 +0000 UTC m=+1089.569975651" watchObservedRunningTime="2026-11-03 20:15:06.641462773 +0000 UTC m=+1089.573330290" Nov 03 20:15:07.623073 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:07.623036 4918 generic.go:356] "Generic (PLEG): container finished" podID="28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce" containerID="c85f2497a8d10944c107d56f964f9183781adbc1e2919fad93ff55420eb85f3c" exitCode=0 Nov 03 20:15:07.623502 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:07.623124 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9" event={"ID":"28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce","Type":"ContainerDied","Data":"c85f2497a8d10944c107d56f964f9183781adbc1e2919fad93ff55420eb85f3c"} Nov 03 20:15:08.836333 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:08.836301 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9" Nov 03 20:15:08.945963 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:08.945866 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-maas-api-xks-ca\") pod \"28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce\" (UID: \"28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce\") " Nov 03 20:15:08.946138 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:08.945992 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-kube-api-access-cdjl4\" (UniqueName: \"kubernetes.io/projected/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-kube-api-access-cdjl4\") pod \"28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce\" (UID: \"28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce\") " Nov 03 20:15:08.946138 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:08.946102 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-maas-api-openshift-ca\") pod \"28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce\" (UID: \"28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce\") " Nov 03 20:15:08.946593 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:08.946558 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-maas-api-openshift-ca" pod "28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce" (UID: "28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:08.946809 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:08.946790 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-maas-api-openshift-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:08.948277 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:08.948253 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-maas-api-xks-ca" pod "28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce" (UID: "28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:08.948596 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:08.948574 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-kube-api-access-cdjl4" pod "28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce" (UID: "28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce"). InnerVolumeSpecName "kube-api-access-cdjl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:09.047952 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:09.047919 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdjl4\" (UniqueName: \"kubernetes.io/projected/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-kube-api-access-cdjl4\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:09.047952 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:09.047947 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce-maas-api-xks-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:09.639070 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:09.639038 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9" Nov 03 20:15:09.639292 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:09.639071 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9" event={"ID":"28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce","Type":"ContainerDied","Data":"ac6c01df1b9265e1ff5ec85b8a322db41866102c9f8eaf125a3f9e383d46ccab"} Nov 03 20:15:09.639292 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:09.639122 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac6c01df1b9265e1ff5ec85b8a322db41866102c9f8eaf125a3f9e383d46ccab" Nov 03 20:15:16.234699 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.234660 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79"] Nov 03 20:15:16.235332 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.235317 4918 cpu_manager.go:414] "containerMap: removing container" podUID="28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce" containerName="revoke-keys" Nov 03 20:15:16.235383 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.235333 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce" containerName="revoke-keys" Nov 03 20:15:16.235442 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.235433 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce" containerName="revoke-keys" Nov 03 20:15:16.239090 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.239067 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.255480 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.255453 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79"] Nov 03 20:15:16.291940 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.291908 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.292100 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.291948 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.292100 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.291977 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.292204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.292105 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.292204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.292163 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8rvn\" (UniqueName: \"kubernetes.io/projected/4e08c21d-00c5-4769-87fc-253cb19a7c85-kube-api-access-h8rvn\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.292454 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.292432 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.292454 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.292463 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e08c21d-00c5-4769-87fc-253cb19a7c85-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.292646 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.292583 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.292747 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.292726 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.394790 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.394752 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.395002 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.394887 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.395084 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.395016 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.395084 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.395043 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.395084 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.395072 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.395292 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.395139 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.395292 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.395215 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8rvn\" (UniqueName: \"kubernetes.io/projected/4e08c21d-00c5-4769-87fc-253cb19a7c85-kube-api-access-h8rvn\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.395292 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.395224 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.395452 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.395293 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.395452 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.395325 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e08c21d-00c5-4769-87fc-253cb19a7c85-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.395558 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.395508 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.395558 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.395526 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.395825 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.395802 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.396295 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.396268 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e08c21d-00c5-4769-87fc-253cb19a7c85-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.397918 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.397868 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.398761 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.398735 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.433423 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.433329 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8rvn\" (UniqueName: \"kubernetes.io/projected/4e08c21d-00c5-4769-87fc-253cb19a7c85-kube-api-access-h8rvn\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.434885 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.434854 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.555026 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.554929 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:16.848800 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.848768 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79"] Nov 03 20:15:16.851007 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:15:16.850978 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e08c21d_00c5_4769_87fc_253cb19a7c85.slice/crio-b71fe7813598caabf6cd458bc4b0c4db4222b53d8299766ae8ab5d0965a49429 WatchSource:0}: Error finding container b71fe7813598caabf6cd458bc4b0c4db4222b53d8299766ae8ab5d0965a49429: Status 404 returned error can't find the container with id b71fe7813598caabf6cd458bc4b0c4db4222b53d8299766ae8ab5d0965a49429 Nov 03 20:15:16.853718 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.853682 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:15:16.853806 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.853752 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:15:16.853806 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:16.853792 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:15:17.749240 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:17.746942 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" event={"ID":"4e08c21d-00c5-4769-87fc-253cb19a7c85","Type":"ContainerStarted","Data":"ab910217d99b91ab013369e3d6df3d63464b36cdb8c168fef4d1b11903e5bdcb"} Nov 03 20:15:17.763539 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:17.758441 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" event={"ID":"4e08c21d-00c5-4769-87fc-253cb19a7c85","Type":"ContainerStarted","Data":"b71fe7813598caabf6cd458bc4b0c4db4222b53d8299766ae8ab5d0965a49429"} Nov 03 20:15:17.859635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:17.859548 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" podStartSLOduration=1.859528263 podStartE2EDuration="1.859528263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:17.857130427 +0000 UTC m=+1100.788997975" watchObservedRunningTime="2026-11-03 20:15:17.859528263 +0000 UTC m=+1100.791395768" Nov 03 20:15:18.555792 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:18.555758 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:18.557407 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:18.557380 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:15:18.557543 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:18.557438 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" podUID="4e08c21d-00c5-4769-87fc-253cb19a7c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 20:15:19.556127 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:19.556091 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:15:19.556634 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:19.556168 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" podUID="4e08c21d-00c5-4769-87fc-253cb19a7c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 20:15:20.556328 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:20.556296 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:15:20.556741 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:20.556358 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" podUID="4e08c21d-00c5-4769-87fc-253cb19a7c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 20:15:21.555708 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:21.555674 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:15:21.555894 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:21.555727 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" podUID="4e08c21d-00c5-4769-87fc-253cb19a7c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 20:15:22.556075 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:22.556027 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:15:22.556555 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:22.556096 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" podUID="4e08c21d-00c5-4769-87fc-253cb19a7c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 20:15:23.556167 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:23.556125 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:15:23.556801 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:23.556238 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" podUID="4e08c21d-00c5-4769-87fc-253cb19a7c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 20:15:25.570370 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:25.570325 4918 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:25.571142 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:25.571106 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" podUID="4e08c21d-00c5-4769-87fc-253cb19a7c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:25.583814 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:25.583781 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:25.585575 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:25.585554 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:25.599200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:25.597432 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:15:26.307338 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:26.307293 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:26.386312 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:26.386263 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:27.312200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:27.309446 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 20:15:27.414202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:27.410963 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:27.800230 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:27.800124 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:27.840412 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:27.823055 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded" Nov 03 20:15:28.000037 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:27.999996 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 20:15:28.001448 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:27.798892 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 20:15:28.027505 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:28.002944 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 20:15:28.217202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:28.215453 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 20:15:28.217202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:28.216481 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:28.217202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:28.216556 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" podUID="f6f6c399-4702-41b4-b4f4-8920d34d7cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:28.513772 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:28.460713 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:28.595330 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:28.595275 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:28.733261 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:15:28.733208 4918 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.496s" Nov 03 20:15:28.733471 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:28.733308 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:15:28.733471 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:28.733380 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:15:28.733471 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:28.733419 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:15:28.733471 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:28.733453 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:15:28.949856 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:28.939450 4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:28.970505 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:28.970464 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" podUID="645fc172-d7a8-48a6-93ec-6c9ed0fc0304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:28.970751 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:28.948823 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:29.264488 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:29.256648 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:29.264488 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:29.257273 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:29.264488 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:29.258866 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:15:29.264488 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:29.258953 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:15:29.264488 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:15:29.261911 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:15:29.272227 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:29.268406 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:15:29.272227 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:29.268509 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:15:29.328195 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:29.282667 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:29.328195 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:29.324563 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:15:29.328195 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:29.324624 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:15:29.344318 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:29.343412 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:15:29.344318 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:29.343839 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:15:29.447345 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:29.447315 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:15:29.448506 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:29.448484 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:15:46.101266 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.101153 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps"] Nov 03 20:15:46.105622 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.105599 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps" Nov 03 20:15:46.116460 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.116441 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c847b846\"" Nov 03 20:15:46.116713 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.116694 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c847b846-dockercfg-fn8gp\"" Nov 03 20:15:46.127366 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.127341 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps"] Nov 03 20:15:46.130853 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.130814 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c62415-141d-43d2-b58f-1855e470d77a-plugins-config-volume\") pod \"payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps\" (UID: \"93c62415-141d-43d2-b58f-1855e470d77a\") " pod="openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps" Nov 03 20:15:46.131010 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.130993 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p4qn\" (UniqueName: \"kubernetes.io/projected/93c62415-141d-43d2-b58f-1855e470d77a-kube-api-access-7p4qn\") pod \"payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps\" (UID: \"93c62415-141d-43d2-b58f-1855e470d77a\") " pod="openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps" Nov 03 20:15:46.131979 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.131956 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj"] Nov 03 20:15:46.137411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.137393 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj" Nov 03 20:15:46.238208 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.235414 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfhsm\" (UniqueName: \"kubernetes.io/projected/4f8943e2-e60e-48dd-a275-ca7fdedbf13e-kube-api-access-rfhsm\") pod \"payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj\" (UID: \"4f8943e2-e60e-48dd-a275-ca7fdedbf13e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj" Nov 03 20:15:46.238208 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.235629 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c62415-141d-43d2-b58f-1855e470d77a-plugins-config-volume\") pod \"payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps\" (UID: \"93c62415-141d-43d2-b58f-1855e470d77a\") " pod="openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps" Nov 03 20:15:46.238208 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.235731 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8943e2-e60e-48dd-a275-ca7fdedbf13e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj\" (UID: \"4f8943e2-e60e-48dd-a275-ca7fdedbf13e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj" Nov 03 20:15:46.238208 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.235772 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p4qn\" (UniqueName: \"kubernetes.io/projected/93c62415-141d-43d2-b58f-1855e470d77a-kube-api-access-7p4qn\") pod \"payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps\" (UID: \"93c62415-141d-43d2-b58f-1855e470d77a\") " pod="openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps" Nov 03 20:15:46.238208 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.237099 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c62415-141d-43d2-b58f-1855e470d77a-plugins-config-volume\") pod \"payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps\" (UID: \"93c62415-141d-43d2-b58f-1855e470d77a\") " pod="openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps" Nov 03 20:15:46.301436 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.292715 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj"] Nov 03 20:15:46.307342 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.307310 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p4qn\" (UniqueName: \"kubernetes.io/projected/93c62415-141d-43d2-b58f-1855e470d77a-kube-api-access-7p4qn\") pod \"payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps\" (UID: \"93c62415-141d-43d2-b58f-1855e470d77a\") " pod="openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps" Nov 03 20:15:46.340202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.338264 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8943e2-e60e-48dd-a275-ca7fdedbf13e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj\" (UID: \"4f8943e2-e60e-48dd-a275-ca7fdedbf13e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj" Nov 03 20:15:46.340202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.338506 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhsm\" (UniqueName: \"kubernetes.io/projected/4f8943e2-e60e-48dd-a275-ca7fdedbf13e-kube-api-access-rfhsm\") pod \"payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj\" (UID: \"4f8943e2-e60e-48dd-a275-ca7fdedbf13e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj" Nov 03 20:15:46.347798 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.341771 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8943e2-e60e-48dd-a275-ca7fdedbf13e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj\" (UID: \"4f8943e2-e60e-48dd-a275-ca7fdedbf13e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj" Nov 03 20:15:46.361276 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.360651 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfhsm\" (UniqueName: \"kubernetes.io/projected/4f8943e2-e60e-48dd-a275-ca7fdedbf13e-kube-api-access-rfhsm\") pod \"payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj\" (UID: \"4f8943e2-e60e-48dd-a275-ca7fdedbf13e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj" Nov 03 20:15:46.420297 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.420249 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps" Nov 03 20:15:46.588190 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.588140 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj" Nov 03 20:15:46.747594 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.747551 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps"] Nov 03 20:15:46.747803 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:15:46.747781 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c62415_141d_43d2_b58f_1855e470d77a.slice/crio-094b2f7759bdf0d0f6b0150c1be79d09fbfeff9a137eacdf4a510d20d1730702 WatchSource:0}: Error finding container 094b2f7759bdf0d0f6b0150c1be79d09fbfeff9a137eacdf4a510d20d1730702: Status 404 returned error can't find the container with id 094b2f7759bdf0d0f6b0150c1be79d09fbfeff9a137eacdf4a510d20d1730702 Nov 03 20:15:46.861859 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.861820 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps" event={"ID":"93c62415-141d-43d2-b58f-1855e470d77a","Type":"ContainerStarted","Data":"5bc9f3968e08defd20c4e7e29fff0e277e42271c8e2dad4b16448e669ed6c5b5"} Nov 03 20:15:46.862007 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.861869 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps" event={"ID":"93c62415-141d-43d2-b58f-1855e470d77a","Type":"ContainerStarted","Data":"094b2f7759bdf0d0f6b0150c1be79d09fbfeff9a137eacdf4a510d20d1730702"} Nov 03 20:15:46.862055 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.862004 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps" Nov 03 20:15:46.888212 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.888143 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps" podStartSLOduration=0.888127223 podStartE2EDuration="888.127223ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:46.88568337 +0000 UTC m=+1129.817550882" watchObservedRunningTime="2026-11-03 20:15:46.888127223 +0000 UTC m=+1129.819994738" Nov 03 20:15:46.899467 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:46.899438 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj"] Nov 03 20:15:46.908768 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:15:46.908746 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8943e2_e60e_48dd_a275_ca7fdedbf13e.slice/crio-2440777f6d06ebe9ef3a8aab1a8af3d8451d8f7c1dd79a0dc905f872bd69bd25 WatchSource:0}: Error finding container 2440777f6d06ebe9ef3a8aab1a8af3d8451d8f7c1dd79a0dc905f872bd69bd25: Status 404 returned error can't find the container with id 2440777f6d06ebe9ef3a8aab1a8af3d8451d8f7c1dd79a0dc905f872bd69bd25 Nov 03 20:15:47.869793 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:47.869759 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj" event={"ID":"4f8943e2-e60e-48dd-a275-ca7fdedbf13e","Type":"ContainerStarted","Data":"b5e5f3996e6bf10222783024ae2302fbf5a9da1e7b129e14d431a5234372bc36"} Nov 03 20:15:47.870248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:47.869845 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj" event={"ID":"4f8943e2-e60e-48dd-a275-ca7fdedbf13e","Type":"ContainerStarted","Data":"2440777f6d06ebe9ef3a8aab1a8af3d8451d8f7c1dd79a0dc905f872bd69bd25"} Nov 03 20:15:47.870248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:47.869877 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj" Nov 03 20:15:47.891226 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:47.891157 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj" podStartSLOduration=1.891144128 podStartE2EDuration="1.891144128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:47.887252222 +0000 UTC m=+1130.819119733" watchObservedRunningTime="2026-11-03 20:15:47.891144128 +0000 UTC m=+1130.823011638" Nov 03 20:15:50.739533 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:50.739497 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t"] Nov 03 20:15:50.744437 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:50.744417 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t" Nov 03 20:15:50.752793 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:50.752771 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e44e2278\"" Nov 03 20:15:50.755211 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:50.754365 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e44e2278-dockercfg-6qkzk\"" Nov 03 20:15:50.755211 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:50.754865 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t"] Nov 03 20:15:50.759099 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:50.759077 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9"] Nov 03 20:15:50.764053 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:50.764031 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9" Nov 03 20:15:50.783417 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:50.783387 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9"] Nov 03 20:15:50.891202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:50.889998 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb2bk\" (UniqueName: \"kubernetes.io/projected/b725b1ec-66c1-497f-b04b-4d59cc455dad-kube-api-access-xb2bk\") pod \"payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9\" (UID: \"b725b1ec-66c1-497f-b04b-4d59cc455dad\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9" Nov 03 20:15:50.891202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:50.890230 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b725b1ec-66c1-497f-b04b-4d59cc455dad-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9\" (UID: \"b725b1ec-66c1-497f-b04b-4d59cc455dad\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9" Nov 03 20:15:50.920203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:50.914266 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d196f95-8e91-4afc-a96f-5716f32265ed-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t\" (UID: \"9d196f95-8e91-4afc-a96f-5716f32265ed\") " pod="openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t" Nov 03 20:15:50.920203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:50.914309 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvgxq\" (UniqueName: \"kubernetes.io/projected/9d196f95-8e91-4afc-a96f-5716f32265ed-kube-api-access-mvgxq\") pod \"payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t\" (UID: \"9d196f95-8e91-4afc-a96f-5716f32265ed\") " pod="openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t" Nov 03 20:15:51.039389 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:51.031217 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b725b1ec-66c1-497f-b04b-4d59cc455dad-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9\" (UID: \"b725b1ec-66c1-497f-b04b-4d59cc455dad\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9" Nov 03 20:15:51.039389 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:51.031318 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d196f95-8e91-4afc-a96f-5716f32265ed-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t\" (UID: \"9d196f95-8e91-4afc-a96f-5716f32265ed\") " pod="openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t" Nov 03 20:15:51.039389 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:51.031346 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvgxq\" (UniqueName: \"kubernetes.io/projected/9d196f95-8e91-4afc-a96f-5716f32265ed-kube-api-access-mvgxq\") pod \"payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t\" (UID: \"9d196f95-8e91-4afc-a96f-5716f32265ed\") " pod="openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t" Nov 03 20:15:51.039389 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:51.031530 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb2bk\" (UniqueName: \"kubernetes.io/projected/b725b1ec-66c1-497f-b04b-4d59cc455dad-kube-api-access-xb2bk\") pod \"payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9\" (UID: \"b725b1ec-66c1-497f-b04b-4d59cc455dad\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9" Nov 03 20:15:51.039389 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:51.032825 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b725b1ec-66c1-497f-b04b-4d59cc455dad-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9\" (UID: \"b725b1ec-66c1-497f-b04b-4d59cc455dad\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9" Nov 03 20:15:51.039389 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:51.033544 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d196f95-8e91-4afc-a96f-5716f32265ed-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t\" (UID: \"9d196f95-8e91-4afc-a96f-5716f32265ed\") " pod="openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t" Nov 03 20:15:51.050691 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:51.050658 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb2bk\" (UniqueName: \"kubernetes.io/projected/b725b1ec-66c1-497f-b04b-4d59cc455dad-kube-api-access-xb2bk\") pod \"payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9\" (UID: \"b725b1ec-66c1-497f-b04b-4d59cc455dad\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9" Nov 03 20:15:51.051029 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:51.051010 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvgxq\" (UniqueName: \"kubernetes.io/projected/9d196f95-8e91-4afc-a96f-5716f32265ed-kube-api-access-mvgxq\") pod \"payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t\" (UID: \"9d196f95-8e91-4afc-a96f-5716f32265ed\") " pod="openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t" Nov 03 20:15:51.061693 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:51.061661 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t" Nov 03 20:15:51.089332 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:51.089297 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9" Nov 03 20:15:51.353986 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:51.353954 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t"] Nov 03 20:15:51.417080 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:51.417049 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9"] Nov 03 20:15:51.418456 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:15:51.418426 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb725b1ec_66c1_497f_b04b_4d59cc455dad.slice/crio-8c5023213dfd2dbe84c2b870a19f00ca1d633827906a48c1b079957386acd0d7 WatchSource:0}: Error finding container 8c5023213dfd2dbe84c2b870a19f00ca1d633827906a48c1b079957386acd0d7: Status 404 returned error can't find the container with id 8c5023213dfd2dbe84c2b870a19f00ca1d633827906a48c1b079957386acd0d7 Nov 03 20:15:52.005836 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:52.005787 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9" event={"ID":"b725b1ec-66c1-497f-b04b-4d59cc455dad","Type":"ContainerStarted","Data":"4ce82d630478e7411db0bcafef9c07c4e52ec5cda92a663e1fb0f0e4c79d59eb"} Nov 03 20:15:52.006355 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:52.005866 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9" event={"ID":"b725b1ec-66c1-497f-b04b-4d59cc455dad","Type":"ContainerStarted","Data":"8c5023213dfd2dbe84c2b870a19f00ca1d633827906a48c1b079957386acd0d7"} Nov 03 20:15:52.006355 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:52.006015 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9" Nov 03 20:15:52.008121 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:52.008092 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t" event={"ID":"9d196f95-8e91-4afc-a96f-5716f32265ed","Type":"ContainerStarted","Data":"dfe55da043a447cd79c6013a7e82d6f20933e56e820bfa6fdd315943d4f9e2d2"} Nov 03 20:15:52.008280 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:52.008136 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t" event={"ID":"9d196f95-8e91-4afc-a96f-5716f32265ed","Type":"ContainerStarted","Data":"4b4a888937d9d7fb6bf8144f06fe8c9414aab8ad8f851fa7d89b3f4d1e7c1c39"} Nov 03 20:15:52.008280 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:52.008157 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t" Nov 03 20:15:52.026997 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:52.026947 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9" podStartSLOduration=2.026931383 podStartE2EDuration="2.026931383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:52.023948494 +0000 UTC m=+1134.955816047" watchObservedRunningTime="2026-11-03 20:15:52.026931383 +0000 UTC m=+1134.958798884" Nov 03 20:15:52.043370 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:52.043305 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t" podStartSLOduration=2.043288461 podStartE2EDuration="2.043288461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:52.042840498 +0000 UTC m=+1134.974707998" watchObservedRunningTime="2026-11-03 20:15:52.043288461 +0000 UTC m=+1134.975155973" Nov 03 20:15:57.887201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:57.884389 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps" Nov 03 20:15:58.280547 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:58.265882 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:58.320216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:58.308011 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:58.443532 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:58.382580 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:58.443532 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:58.443278 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:58.512327 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:58.509248 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 20:15:58.537275 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:58.518535 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:58.753429 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:15:58.753386 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:15:58.891202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:58.866985 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:58.891202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:58.867077 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" podUID="eba28d4c-a907-4b6f-8d20-0a1baade4c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:59.000204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:58.997992 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:59.330201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:59.329804 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:59.445115 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:59.363371 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:59.445115 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:59.387250 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:59.458200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:59.446116 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:59.514203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:59.511657 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:59.514203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:59.513394 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:59.520524 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:59.520473 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:59.998782 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:59.990035 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:15:59.998782 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:59.990095 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:15:59.998782 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:59.990133 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj" Nov 03 20:15:59.998782 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:59.990198 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:15:59.998782 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:59.990231 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:15:59.998782 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:59.990308 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:15:59.998782 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:15:59.990340 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:16:00.502300 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:00.502256 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 20:16:00.517513 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:00.517472 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:00.519461 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:00.519430 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 20:16:00.552204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:00.526217 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded" Nov 03 20:16:01.008105 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:00.526282 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:16:01.008105 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:00.580307 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 20:16:01.008105 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:00.620426 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:16:01.008105 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:00.937535 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:16:01.008105 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:00.937577 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:16:01.008105 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:00.632959 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 20:16:01.008105 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:00.937614 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:16:01.008105 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:00.964772 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:16:01.008105 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:00.974678 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" Nov 03 20:16:01.008105 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:00.987985 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:16:01.030811 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:01.030781 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:16:03.026631 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:03.026601 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9" Nov 03 20:16:03.027191 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:03.026751 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t" Nov 03 20:16:25.855464 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:25.855430 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps"] Nov 03 20:16:25.868949 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:25.868901 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj"] Nov 03 20:16:28.216206 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:16:28.216135 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:16:29.108336 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.108295 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf"] Nov 03 20:16:29.115403 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.115377 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.125907 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.125877 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf"] Nov 03 20:16:29.156609 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.149597 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ffd1cf5a-openshift-default-dockercfg-l9w2t\"" Nov 03 20:16:29.217074 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.213156 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-workload-certs\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.217074 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.213358 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdwd9\" (UniqueName: \"kubernetes.io/projected/c878724e-bf1c-4ff5-836b-af4e0304c6dc-kube-api-access-rdwd9\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.217074 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.213405 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-workload-socket\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.217074 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.213437 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-envoy\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.217074 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.213471 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-data\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.217074 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.213507 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istiod-ca-cert\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.217074 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.213664 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-token\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.217074 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.213764 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-podinfo\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.217074 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.213836 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-credential-socket\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.322209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.320790 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-workload-certs\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.322209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.320930 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdwd9\" (UniqueName: \"kubernetes.io/projected/c878724e-bf1c-4ff5-836b-af4e0304c6dc-kube-api-access-rdwd9\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.322209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.320986 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-workload-socket\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.322209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.321033 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-envoy\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.322209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.321068 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-data\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.322209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.321112 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istiod-ca-cert\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.322209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.321279 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-token\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.322209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.321367 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-podinfo\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.322209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.321429 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-credential-socket\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.322209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.321956 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-credential-socket\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.322879 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.322290 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-workload-certs\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.322943 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.322919 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-workload-socket\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.333693 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.323752 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-data\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.333693 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.324143 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istiod-ca-cert\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.390416 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.390333 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdwd9\" (UniqueName: \"kubernetes.io/projected/c878724e-bf1c-4ff5-836b-af4e0304c6dc-kube-api-access-rdwd9\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.440152 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.440102 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-envoy\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.463345 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.461438 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-token\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.463345 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.462657 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-podinfo\") pod \"e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:29.761261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:29.760483 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:30.415847 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:30.415773 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf"] Nov 03 20:16:30.583566 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:30.583518 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:16:30.583795 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:30.583768 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:16:30.583955 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:30.583933 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:16:30.863168 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:30.855711 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps" podUID="93c62415-141d-43d2-b58f-1855e470d77a" containerName="payload-processing" containerID="cri-o://5bc9f3968e08defd20c4e7e29fff0e277e42271c8e2dad4b16448e669ed6c5b5" gracePeriod=25 Nov 03 20:16:30.906808 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:30.906280 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj" podUID="4f8943e2-e60e-48dd-a275-ca7fdedbf13e" containerName="payload-pre-processing" containerID="cri-o://b5e5f3996e6bf10222783024ae2302fbf5a9da1e7b129e14d431a5234372bc36" gracePeriod=25 Nov 03 20:16:31.295490 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:31.295454 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t"] Nov 03 20:16:31.425211 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:31.404203 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9"] Nov 03 20:16:31.537282 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:31.534937 4918 generic.go:356] "Generic (PLEG): container finished" podID="93c62415-141d-43d2-b58f-1855e470d77a" containerID="5bc9f3968e08defd20c4e7e29fff0e277e42271c8e2dad4b16448e669ed6c5b5" exitCode=0 Nov 03 20:16:31.537282 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:31.535103 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps" event={"ID":"93c62415-141d-43d2-b58f-1855e470d77a","Type":"ContainerDied","Data":"5bc9f3968e08defd20c4e7e29fff0e277e42271c8e2dad4b16448e669ed6c5b5"} Nov 03 20:16:31.611202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:31.611031 4918 generic.go:356] "Generic (PLEG): container finished" podID="4f8943e2-e60e-48dd-a275-ca7fdedbf13e" containerID="b5e5f3996e6bf10222783024ae2302fbf5a9da1e7b129e14d431a5234372bc36" exitCode=0 Nov 03 20:16:31.611409 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:31.611272 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj" event={"ID":"4f8943e2-e60e-48dd-a275-ca7fdedbf13e","Type":"ContainerDied","Data":"b5e5f3996e6bf10222783024ae2302fbf5a9da1e7b129e14d431a5234372bc36"} Nov 03 20:16:31.672201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:31.656755 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" event={"ID":"c878724e-bf1c-4ff5-836b-af4e0304c6dc","Type":"ContainerStarted","Data":"e9a1d2237222b859f6f800faca16bb42f9b604e85ac6ea529af4dea003228638"} Nov 03 20:16:31.672201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:31.656834 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" event={"ID":"c878724e-bf1c-4ff5-836b-af4e0304c6dc","Type":"ContainerStarted","Data":"bfa613742d67ea6f0e486c3a98b70463e62aa2d83a88516b981c5000d8a57b12"} Nov 03 20:16:31.798290 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:31.783560 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:31.798680 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:31.797460 4918 patch_prober.go:29] interesting pod/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:16:31.798882 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:31.798854 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" podUID="c878724e-bf1c-4ff5-836b-af4e0304c6dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 20:16:32.132202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:32.131688 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:32.259708 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:32.219750 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:32.259708 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:32.229634 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" podStartSLOduration=3.229613381 podStartE2EDuration="3.229613381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:31.797913264 +0000 UTC m=+1174.729780800" watchObservedRunningTime="2026-11-03 20:16:32.229613381 +0000 UTC m=+1175.161480893" Nov 03 20:16:32.457200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:32.457100 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps" Nov 03 20:16:32.613201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:32.595015 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c62415-141d-43d2-b58f-1855e470d77a-kube-api-access-7p4qn\" (UniqueName: \"kubernetes.io/projected/93c62415-141d-43d2-b58f-1855e470d77a-kube-api-access-7p4qn\") pod \"93c62415-141d-43d2-b58f-1855e470d77a\" (UID: \"93c62415-141d-43d2-b58f-1855e470d77a\") " Nov 03 20:16:32.613201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:32.595140 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c62415-141d-43d2-b58f-1855e470d77a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c62415-141d-43d2-b58f-1855e470d77a-plugins-config-volume\") pod \"93c62415-141d-43d2-b58f-1855e470d77a\" (UID: \"93c62415-141d-43d2-b58f-1855e470d77a\") " Nov 03 20:16:32.613201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:32.600249 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 20:16:32.613201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:32.607385 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c62415-141d-43d2-b58f-1855e470d77a-plugins-config-volume" pod "93c62415-141d-43d2-b58f-1855e470d77a" (UID: "93c62415-141d-43d2-b58f-1855e470d77a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:32.696483 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:32.696436 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c62415-141d-43d2-b58f-1855e470d77a-kube-api-access-7p4qn" pod "93c62415-141d-43d2-b58f-1855e470d77a" (UID: "93c62415-141d-43d2-b58f-1855e470d77a"). InnerVolumeSpecName "kube-api-access-7p4qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:32.866231 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:32.849011 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p4qn\" (UniqueName: \"kubernetes.io/projected/93c62415-141d-43d2-b58f-1855e470d77a-kube-api-access-7p4qn\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:32.866231 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:32.849045 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c62415-141d-43d2-b58f-1855e470d77a-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:32.901964 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:32.901921 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:33.094747 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:33.094709 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps" Nov 03 20:16:33.095517 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:33.095294 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps" event={"ID":"93c62415-141d-43d2-b58f-1855e470d77a","Type":"ContainerDied","Data":"094b2f7759bdf0d0f6b0150c1be79d09fbfeff9a137eacdf4a510d20d1730702"} Nov 03 20:16:33.095517 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:33.095364 4918 scope.go:122] "RemoveContainer" containerID="5bc9f3968e08defd20c4e7e29fff0e277e42271c8e2dad4b16448e669ed6c5b5" Nov 03 20:16:33.097245 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:33.096605 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:33.253578 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:33.253537 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:33.509257 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:33.495323 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj" Nov 03 20:16:33.639201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:33.638772 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps"] Nov 03 20:16:33.657200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:33.639773 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f8943e2-e60e-48dd-a275-ca7fdedbf13e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8943e2-e60e-48dd-a275-ca7fdedbf13e-plugins-config-volume\") pod \"4f8943e2-e60e-48dd-a275-ca7fdedbf13e\" (UID: \"4f8943e2-e60e-48dd-a275-ca7fdedbf13e\") " Nov 03 20:16:33.657200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:33.640241 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f8943e2-e60e-48dd-a275-ca7fdedbf13e-kube-api-access-rfhsm\" (UniqueName: \"kubernetes.io/projected/4f8943e2-e60e-48dd-a275-ca7fdedbf13e-kube-api-access-rfhsm\") pod \"4f8943e2-e60e-48dd-a275-ca7fdedbf13e\" (UID: \"4f8943e2-e60e-48dd-a275-ca7fdedbf13e\") " Nov 03 20:16:33.657200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:33.654169 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f8943e2-e60e-48dd-a275-ca7fdedbf13e-plugins-config-volume" pod "4f8943e2-e60e-48dd-a275-ca7fdedbf13e" (UID: "4f8943e2-e60e-48dd-a275-ca7fdedbf13e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:33.786303 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:33.783465 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8943e2-e60e-48dd-a275-ca7fdedbf13e-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:33.864214 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:33.857462 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8943e2-e60e-48dd-a275-ca7fdedbf13e-kube-api-access-rfhsm" pod "4f8943e2-e60e-48dd-a275-ca7fdedbf13e" (UID: "4f8943e2-e60e-48dd-a275-ca7fdedbf13e"). InnerVolumeSpecName "kube-api-access-rfhsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:33.864214 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:33.857543 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c847b846-8588b8cf74-j5nps"] Nov 03 20:16:34.012199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.012059 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfhsm\" (UniqueName: \"kubernetes.io/projected/4f8943e2-e60e-48dd-a275-ca7fdedbf13e-kube-api-access-rfhsm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:34.283840 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.283808 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj" Nov 03 20:16:34.313210 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.294597 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj" event={"ID":"4f8943e2-e60e-48dd-a275-ca7fdedbf13e","Type":"ContainerDied","Data":"2440777f6d06ebe9ef3a8aab1a8af3d8451d8f7c1dd79a0dc905f872bd69bd25"} Nov 03 20:16:34.313210 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.294733 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:34.313210 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.294774 4918 scope.go:122] "RemoveContainer" containerID="b5e5f3996e6bf10222783024ae2302fbf5a9da1e7b129e14d431a5234372bc36" Nov 03 20:16:34.313210 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.308904 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:16:34.314882 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.314850 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb"] Nov 03 20:16:34.343581 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.315917 4918 cpu_manager.go:414] "containerMap: removing container" podUID="4f8943e2-e60e-48dd-a275-ca7fdedbf13e" containerName="payload-pre-processing" Nov 03 20:16:34.343581 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.315941 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f8943e2-e60e-48dd-a275-ca7fdedbf13e" containerName="payload-pre-processing" Nov 03 20:16:34.343581 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.315973 4918 cpu_manager.go:414] "containerMap: removing container" podUID="93c62415-141d-43d2-b58f-1855e470d77a" containerName="payload-processing" Nov 03 20:16:34.343581 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.315982 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c62415-141d-43d2-b58f-1855e470d77a" containerName="payload-processing" Nov 03 20:16:34.343581 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.316164 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f8943e2-e60e-48dd-a275-ca7fdedbf13e" containerName="payload-pre-processing" Nov 03 20:16:34.343581 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.316216 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c62415-141d-43d2-b58f-1855e470d77a" containerName="payload-processing" Nov 03 20:16:34.343581 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.326541 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.343581 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.334367 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-11b0d340-openshift-default-dockercfg-g42n7\"" Nov 03 20:16:34.350753 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.350721 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-workload-certs\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.359994 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.359971 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-workload-socket\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.361168 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.361145 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-data\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.361467 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.361445 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7tkt\" (UniqueName: \"kubernetes.io/projected/46b16107-a3b5-46fe-a4be-7fde061ac3da-kube-api-access-f7tkt\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.362094 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.362074 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b16107-a3b5-46fe-a4be-7fde061ac3da-istiod-ca-cert\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.362364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.362341 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-podinfo\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.362817 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.362798 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-token\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.362984 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.362965 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-envoy\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.363225 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.363171 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-credential-socket\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.417266 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.417234 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb"] Nov 03 20:16:34.462500 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.462465 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj"] Nov 03 20:16:34.462745 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.462725 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c847b846-bd8547ccf-cj5tj"] Nov 03 20:16:34.465361 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.465315 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-data\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.466207 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.466067 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-data\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.466872 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.466833 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7tkt\" (UniqueName: \"kubernetes.io/projected/46b16107-a3b5-46fe-a4be-7fde061ac3da-kube-api-access-f7tkt\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.467077 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.467057 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b16107-a3b5-46fe-a4be-7fde061ac3da-istiod-ca-cert\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.468043 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.468023 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-podinfo\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.468685 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.468652 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-token\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.469127 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.469098 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-envoy\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.469301 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.469283 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-credential-socket\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.469492 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.469472 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-workload-certs\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.470385 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.470360 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-workload-socket\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.470961 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.469857 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-credential-socket\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.471198 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.470066 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-workload-certs\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.471334 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.467943 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b16107-a3b5-46fe-a4be-7fde061ac3da-istiod-ca-cert\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.471493 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.470856 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-workload-socket\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.645286 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.604353 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-token\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.645286 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.605050 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-envoy\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.645286 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.631712 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-podinfo\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.654575 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.654103 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7tkt\" (UniqueName: \"kubernetes.io/projected/46b16107-a3b5-46fe-a4be-7fde061ac3da-kube-api-access-f7tkt\") pod \"e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.727090 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.722550 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:34.940573 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.940481 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 20:16:34.970200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.967258 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8bvn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:34.970200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:34.967354 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" podUID="ea819fe0-461b-4952-a40e-7a47e8f51580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:35.239974 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:35.233663 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f8943e2-e60e-48dd-a275-ca7fdedbf13e" path="/var/lib/kubelet/pods/4f8943e2-e60e-48dd-a275-ca7fdedbf13e/volumes" Nov 03 20:16:35.241678 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:35.241401 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c62415-141d-43d2-b58f-1855e470d77a" path="/var/lib/kubelet/pods/93c62415-141d-43d2-b58f-1855e470d77a/volumes" Nov 03 20:16:35.388589 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:35.388558 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb"] Nov 03 20:16:35.392783 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:35.391590 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:16:35.392783 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:35.391676 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:16:35.392783 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:35.391724 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:16:36.296048 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:36.295999 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t" podUID="9d196f95-8e91-4afc-a96f-5716f32265ed" containerName="payload-processing" containerID="cri-o://dfe55da043a447cd79c6013a7e82d6f20933e56e820bfa6fdd315943d4f9e2d2" gracePeriod=25 Nov 03 20:16:36.317572 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:36.317533 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" event={"ID":"46b16107-a3b5-46fe-a4be-7fde061ac3da","Type":"ContainerStarted","Data":"17951a0f014581820e8bfc4f4843c9e54d59052add312ec9061e56d8c5f4d5fc"} Nov 03 20:16:36.317740 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:36.317605 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" event={"ID":"46b16107-a3b5-46fe-a4be-7fde061ac3da","Type":"ContainerStarted","Data":"8ae0a8b7b8161d50fbf6bf6796c00c917c8c6620700c3037f76d8fa5744e2db8"} Nov 03 20:16:36.343685 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:36.343637 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" podStartSLOduration=2.343618681 podStartE2EDuration="2.343618681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:36.34301535 +0000 UTC m=+1179.274882862" watchObservedRunningTime="2026-11-03 20:16:36.343618681 +0000 UTC m=+1179.275486188" Nov 03 20:16:36.404961 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:36.404906 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9" podUID="b725b1ec-66c1-497f-b04b-4d59cc455dad" containerName="payload-pre-processing" containerID="cri-o://4ce82d630478e7411db0bcafef9c07c4e52ec5cda92a663e1fb0f0e4c79d59eb" gracePeriod=25 Nov 03 20:16:36.678830 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:36.678359 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t" Nov 03 20:16:36.707654 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:36.706453 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d196f95-8e91-4afc-a96f-5716f32265ed-kube-api-access-mvgxq\" (UniqueName: \"kubernetes.io/projected/9d196f95-8e91-4afc-a96f-5716f32265ed-kube-api-access-mvgxq\") pod \"9d196f95-8e91-4afc-a96f-5716f32265ed\" (UID: \"9d196f95-8e91-4afc-a96f-5716f32265ed\") " Nov 03 20:16:36.707654 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:36.706771 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d196f95-8e91-4afc-a96f-5716f32265ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d196f95-8e91-4afc-a96f-5716f32265ed-plugins-config-volume\") pod \"9d196f95-8e91-4afc-a96f-5716f32265ed\" (UID: \"9d196f95-8e91-4afc-a96f-5716f32265ed\") " Nov 03 20:16:36.708129 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:36.708071 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d196f95-8e91-4afc-a96f-5716f32265ed-plugins-config-volume" pod "9d196f95-8e91-4afc-a96f-5716f32265ed" (UID: "9d196f95-8e91-4afc-a96f-5716f32265ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:36.709444 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:36.709417 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d196f95-8e91-4afc-a96f-5716f32265ed-kube-api-access-mvgxq" pod "9d196f95-8e91-4afc-a96f-5716f32265ed" (UID: "9d196f95-8e91-4afc-a96f-5716f32265ed"). InnerVolumeSpecName "kube-api-access-mvgxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:36.729706 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:36.729555 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:36.738198 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:36.737303 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:36.809337 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:36.809256 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvgxq\" (UniqueName: \"kubernetes.io/projected/9d196f95-8e91-4afc-a96f-5716f32265ed-kube-api-access-mvgxq\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:36.809337 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:36.809290 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d196f95-8e91-4afc-a96f-5716f32265ed-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:36.874803 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:36.874737 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9" Nov 03 20:16:36.910985 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:36.910950 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b725b1ec-66c1-497f-b04b-4d59cc455dad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b725b1ec-66c1-497f-b04b-4d59cc455dad-plugins-config-volume\") pod \"b725b1ec-66c1-497f-b04b-4d59cc455dad\" (UID: \"b725b1ec-66c1-497f-b04b-4d59cc455dad\") " Nov 03 20:16:36.911414 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:36.911388 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b725b1ec-66c1-497f-b04b-4d59cc455dad-kube-api-access-xb2bk\" (UniqueName: \"kubernetes.io/projected/b725b1ec-66c1-497f-b04b-4d59cc455dad-kube-api-access-xb2bk\") pod \"b725b1ec-66c1-497f-b04b-4d59cc455dad\" (UID: \"b725b1ec-66c1-497f-b04b-4d59cc455dad\") " Nov 03 20:16:36.911768 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:36.911740 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b725b1ec-66c1-497f-b04b-4d59cc455dad-plugins-config-volume" pod "b725b1ec-66c1-497f-b04b-4d59cc455dad" (UID: "b725b1ec-66c1-497f-b04b-4d59cc455dad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:36.912258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:36.912236 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b725b1ec-66c1-497f-b04b-4d59cc455dad-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:36.914206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:36.914165 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b725b1ec-66c1-497f-b04b-4d59cc455dad-kube-api-access-xb2bk" pod "b725b1ec-66c1-497f-b04b-4d59cc455dad" (UID: "b725b1ec-66c1-497f-b04b-4d59cc455dad"). InnerVolumeSpecName "kube-api-access-xb2bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:37.014551 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.014515 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb2bk\" (UniqueName: \"kubernetes.io/projected/b725b1ec-66c1-497f-b04b-4d59cc455dad-kube-api-access-xb2bk\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:37.156163 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.156086 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d"] Nov 03 20:16:37.156770 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.156755 4918 cpu_manager.go:414] "containerMap: removing container" podUID="9d196f95-8e91-4afc-a96f-5716f32265ed" containerName="payload-processing" Nov 03 20:16:37.156818 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.156772 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d196f95-8e91-4afc-a96f-5716f32265ed" containerName="payload-processing" Nov 03 20:16:37.156818 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.156786 4918 cpu_manager.go:414] "containerMap: removing container" podUID="b725b1ec-66c1-497f-b04b-4d59cc455dad" containerName="payload-pre-processing" Nov 03 20:16:37.156818 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.156791 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b725b1ec-66c1-497f-b04b-4d59cc455dad" containerName="payload-pre-processing" Nov 03 20:16:37.156913 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.156900 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d196f95-8e91-4afc-a96f-5716f32265ed" containerName="payload-processing" Nov 03 20:16:37.156913 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.156908 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b725b1ec-66c1-497f-b04b-4d59cc455dad" containerName="payload-pre-processing" Nov 03 20:16:37.160736 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.160718 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d" Nov 03 20:16:37.169904 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.169879 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ffd1cf5a\"" Nov 03 20:16:37.172885 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.172849 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d"] Nov 03 20:16:37.217779 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.217737 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e9bbf6-e6cd-47c2-9761-d5b0aff056f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d\" (UID: \"84e9bbf6-e6cd-47c2-9761-d5b0aff056f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d" Nov 03 20:16:37.217959 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.217800 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c5vb\" (UniqueName: \"kubernetes.io/projected/84e9bbf6-e6cd-47c2-9761-d5b0aff056f0-kube-api-access-7c5vb\") pod \"payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d\" (UID: \"84e9bbf6-e6cd-47c2-9761-d5b0aff056f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d" Nov 03 20:16:37.344670 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.344630 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e9bbf6-e6cd-47c2-9761-d5b0aff056f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d\" (UID: \"84e9bbf6-e6cd-47c2-9761-d5b0aff056f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d" Nov 03 20:16:37.377158 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.377113 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c5vb\" (UniqueName: \"kubernetes.io/projected/84e9bbf6-e6cd-47c2-9761-d5b0aff056f0-kube-api-access-7c5vb\") pod \"payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d\" (UID: \"84e9bbf6-e6cd-47c2-9761-d5b0aff056f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d" Nov 03 20:16:37.385465 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.355520 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv"] Nov 03 20:16:37.391619 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.351973 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e9bbf6-e6cd-47c2-9761-d5b0aff056f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d\" (UID: \"84e9bbf6-e6cd-47c2-9761-d5b0aff056f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d" Nov 03 20:16:37.391873 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.384378 4918 generic.go:356] "Generic (PLEG): container finished" podID="b725b1ec-66c1-497f-b04b-4d59cc455dad" containerID="4ce82d630478e7411db0bcafef9c07c4e52ec5cda92a663e1fb0f0e4c79d59eb" exitCode=0 Nov 03 20:16:37.392190 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.384472 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9" Nov 03 20:16:37.399956 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.399922 4918 generic.go:356] "Generic (PLEG): container finished" podID="9d196f95-8e91-4afc-a96f-5716f32265ed" containerID="dfe55da043a447cd79c6013a7e82d6f20933e56e820bfa6fdd315943d4f9e2d2" exitCode=0 Nov 03 20:16:37.403197 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.402416 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t" Nov 03 20:16:37.425463 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.425384 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv"] Nov 03 20:16:37.425685 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.425659 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9" event={"ID":"b725b1ec-66c1-497f-b04b-4d59cc455dad","Type":"ContainerDied","Data":"4ce82d630478e7411db0bcafef9c07c4e52ec5cda92a663e1fb0f0e4c79d59eb"} Nov 03 20:16:37.425789 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.425705 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:37.425789 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.425736 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9" event={"ID":"b725b1ec-66c1-497f-b04b-4d59cc455dad","Type":"ContainerDied","Data":"8c5023213dfd2dbe84c2b870a19f00ca1d633827906a48c1b079957386acd0d7"} Nov 03 20:16:37.425789 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.425753 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t" event={"ID":"9d196f95-8e91-4afc-a96f-5716f32265ed","Type":"ContainerDied","Data":"dfe55da043a447cd79c6013a7e82d6f20933e56e820bfa6fdd315943d4f9e2d2"} Nov 03 20:16:37.425789 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.425771 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t" event={"ID":"9d196f95-8e91-4afc-a96f-5716f32265ed","Type":"ContainerDied","Data":"4b4a888937d9d7fb6bf8144f06fe8c9414aab8ad8f851fa7d89b3f4d1e7c1c39"} Nov 03 20:16:37.425986 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.425822 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:16:37.425986 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.425856 4918 scope.go:122] "RemoveContainer" containerID="4ce82d630478e7411db0bcafef9c07c4e52ec5cda92a663e1fb0f0e4c79d59eb" Nov 03 20:16:37.431236 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.426084 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv" Nov 03 20:16:37.452208 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.450007 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ffd1cf5a-dockercfg-z66d6\"" Nov 03 20:16:37.465626 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.459903 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c5vb\" (UniqueName: \"kubernetes.io/projected/84e9bbf6-e6cd-47c2-9761-d5b0aff056f0-kube-api-access-7c5vb\") pod \"payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d\" (UID: \"84e9bbf6-e6cd-47c2-9761-d5b0aff056f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d" Nov 03 20:16:37.480752 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.480722 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d" Nov 03 20:16:37.497553 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.497516 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9"] Nov 03 20:16:37.536200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.510464 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj85g\" (UniqueName: \"kubernetes.io/projected/8f0392da-1544-488e-a52b-b0a3508b3dc6-kube-api-access-qj85g\") pod \"payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv\" (UID: \"8f0392da-1544-488e-a52b-b0a3508b3dc6\") " pod="openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv" Nov 03 20:16:37.550251 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.546482 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0392da-1544-488e-a52b-b0a3508b3dc6-plugins-config-volume\") pod \"payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv\" (UID: \"8f0392da-1544-488e-a52b-b0a3508b3dc6\") " pod="openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv" Nov 03 20:16:37.585364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.585253 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e44e2278-6c87464f95-nf4l9"] Nov 03 20:16:37.620365 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.619349 4918 scope.go:122] "RemoveContainer" containerID="4ce82d630478e7411db0bcafef9c07c4e52ec5cda92a663e1fb0f0e4c79d59eb" Nov 03 20:16:37.620998 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:16:37.620938 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ce82d630478e7411db0bcafef9c07c4e52ec5cda92a663e1fb0f0e4c79d59eb\": container with ID starting with 4ce82d630478e7411db0bcafef9c07c4e52ec5cda92a663e1fb0f0e4c79d59eb not found: ID does not exist" containerID="4ce82d630478e7411db0bcafef9c07c4e52ec5cda92a663e1fb0f0e4c79d59eb" Nov 03 20:16:37.621339 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.621309 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ce82d630478e7411db0bcafef9c07c4e52ec5cda92a663e1fb0f0e4c79d59eb"} err="failed to get container status \"4ce82d630478e7411db0bcafef9c07c4e52ec5cda92a663e1fb0f0e4c79d59eb\": rpc error: code = NotFound desc = could not find container \"4ce82d630478e7411db0bcafef9c07c4e52ec5cda92a663e1fb0f0e4c79d59eb\": container with ID starting with 4ce82d630478e7411db0bcafef9c07c4e52ec5cda92a663e1fb0f0e4c79d59eb not found: ID does not exist" Nov 03 20:16:37.621472 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.621461 4918 scope.go:122] "RemoveContainer" containerID="dfe55da043a447cd79c6013a7e82d6f20933e56e820bfa6fdd315943d4f9e2d2" Nov 03 20:16:37.657190 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.657128 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t"] Nov 03 20:16:37.663224 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.663170 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e44e2278-6774d56578-hck8t"] Nov 03 20:16:37.681073 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.680982 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj85g\" (UniqueName: \"kubernetes.io/projected/8f0392da-1544-488e-a52b-b0a3508b3dc6-kube-api-access-qj85g\") pod \"payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv\" (UID: \"8f0392da-1544-488e-a52b-b0a3508b3dc6\") " pod="openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv" Nov 03 20:16:37.681958 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.681929 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0392da-1544-488e-a52b-b0a3508b3dc6-plugins-config-volume\") pod \"payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv\" (UID: \"8f0392da-1544-488e-a52b-b0a3508b3dc6\") " pod="openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv" Nov 03 20:16:37.683412 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.683132 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0392da-1544-488e-a52b-b0a3508b3dc6-plugins-config-volume\") pod \"payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv\" (UID: \"8f0392da-1544-488e-a52b-b0a3508b3dc6\") " pod="openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv" Nov 03 20:16:37.694955 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.694928 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj85g\" (UniqueName: \"kubernetes.io/projected/8f0392da-1544-488e-a52b-b0a3508b3dc6-kube-api-access-qj85g\") pod \"payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv\" (UID: \"8f0392da-1544-488e-a52b-b0a3508b3dc6\") " pod="openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv" Nov 03 20:16:37.700374 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.700318 4918 scope.go:122] "RemoveContainer" containerID="dfe55da043a447cd79c6013a7e82d6f20933e56e820bfa6fdd315943d4f9e2d2" Nov 03 20:16:37.700863 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:16:37.700740 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe55da043a447cd79c6013a7e82d6f20933e56e820bfa6fdd315943d4f9e2d2\": container with ID starting with dfe55da043a447cd79c6013a7e82d6f20933e56e820bfa6fdd315943d4f9e2d2 not found: ID does not exist" containerID="dfe55da043a447cd79c6013a7e82d6f20933e56e820bfa6fdd315943d4f9e2d2" Nov 03 20:16:37.700863 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.700776 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe55da043a447cd79c6013a7e82d6f20933e56e820bfa6fdd315943d4f9e2d2"} err="failed to get container status \"dfe55da043a447cd79c6013a7e82d6f20933e56e820bfa6fdd315943d4f9e2d2\": rpc error: code = NotFound desc = could not find container \"dfe55da043a447cd79c6013a7e82d6f20933e56e820bfa6fdd315943d4f9e2d2\": container with ID starting with dfe55da043a447cd79c6013a7e82d6f20933e56e820bfa6fdd315943d4f9e2d2 not found: ID does not exist" Nov 03 20:16:37.782814 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.782778 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv" Nov 03 20:16:37.951894 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:37.951862 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d"] Nov 03 20:16:38.110058 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:38.110025 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv"] Nov 03 20:16:38.110889 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:16:38.110855 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0392da_1544_488e_a52b_b0a3508b3dc6.slice/crio-0813000b747bbfef6e8c4ed905494bf3a45d582e6c9838109f2595c947048cde WatchSource:0}: Error finding container 0813000b747bbfef6e8c4ed905494bf3a45d582e6c9838109f2595c947048cde: Status 404 returned error can't find the container with id 0813000b747bbfef6e8c4ed905494bf3a45d582e6c9838109f2595c947048cde Nov 03 20:16:38.417895 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:38.417853 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv" event={"ID":"8f0392da-1544-488e-a52b-b0a3508b3dc6","Type":"ContainerStarted","Data":"9b672e6c562882545a91c98d4237336e8e6b8ebadefcce80ff2e4798c97b4f58"} Nov 03 20:16:38.418416 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:38.417958 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv" Nov 03 20:16:38.418416 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:38.417987 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv" event={"ID":"8f0392da-1544-488e-a52b-b0a3508b3dc6","Type":"ContainerStarted","Data":"0813000b747bbfef6e8c4ed905494bf3a45d582e6c9838109f2595c947048cde"} Nov 03 20:16:38.421495 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:38.421462 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d" event={"ID":"84e9bbf6-e6cd-47c2-9761-d5b0aff056f0","Type":"ContainerStarted","Data":"95ca591a606d8c6e69ad3819f0c02c80868251f17953682243962efbb776b9e4"} Nov 03 20:16:38.421643 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:38.421521 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d" event={"ID":"84e9bbf6-e6cd-47c2-9761-d5b0aff056f0","Type":"ContainerStarted","Data":"f9d98cda9f5ccc8c03184f6679b0a54dc8aeb22374c42fcdeaec4c07da170864"} Nov 03 20:16:38.421643 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:38.421554 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d" Nov 03 20:16:38.442381 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:38.442333 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv" podStartSLOduration=1.442320227 podStartE2EDuration="1.442320227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:38.43699638 +0000 UTC m=+1181.368863889" watchObservedRunningTime="2026-11-03 20:16:38.442320227 +0000 UTC m=+1181.374187735" Nov 03 20:16:38.462968 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:38.462915 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d" podStartSLOduration=1.462904725 podStartE2EDuration="1.462904725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:38.459134093 +0000 UTC m=+1181.391001604" watchObservedRunningTime="2026-11-03 20:16:38.462904725 +0000 UTC m=+1181.394772249" Nov 03 20:16:39.216951 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:39.216905 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d196f95-8e91-4afc-a96f-5716f32265ed" path="/var/lib/kubelet/pods/9d196f95-8e91-4afc-a96f-5716f32265ed/volumes" Nov 03 20:16:39.217507 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:39.217485 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b725b1ec-66c1-497f-b04b-4d59cc455dad" path="/var/lib/kubelet/pods/b725b1ec-66c1-497f-b04b-4d59cc455dad/volumes" Nov 03 20:16:39.974782 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:39.974743 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f"] Nov 03 20:16:39.984019 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:39.983991 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f" Nov 03 20:16:39.991433 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:39.991406 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-11b0d340\"" Nov 03 20:16:39.999006 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:39.998959 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-11b0d340-dockercfg-9jhjt\"" Nov 03 20:16:39.999006 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:39.999007 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f"] Nov 03 20:16:40.005131 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:40.005106 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5"] Nov 03 20:16:40.019287 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:40.012527 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5" Nov 03 20:16:40.019287 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:40.013586 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5"] Nov 03 20:16:40.033528 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:40.033498 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f\" (UID: \"14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f" Nov 03 20:16:40.033671 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:40.033639 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmz9c\" (UniqueName: \"kubernetes.io/projected/14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3-kube-api-access-pmz9c\") pod \"payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f\" (UID: \"14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f" Nov 03 20:16:40.134905 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:40.134858 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f\" (UID: \"14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f" Nov 03 20:16:40.135086 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:40.134936 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a98544fd-0619-4632-b931-718a2348d907-plugins-config-volume\") pod \"payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5\" (UID: \"a98544fd-0619-4632-b931-718a2348d907\") " pod="openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5" Nov 03 20:16:40.135086 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:40.135075 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmz9c\" (UniqueName: \"kubernetes.io/projected/14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3-kube-api-access-pmz9c\") pod \"payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f\" (UID: \"14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f" Nov 03 20:16:40.135218 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:40.135169 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw8kl\" (UniqueName: \"kubernetes.io/projected/a98544fd-0619-4632-b931-718a2348d907-kube-api-access-vw8kl\") pod \"payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5\" (UID: \"a98544fd-0619-4632-b931-718a2348d907\") " pod="openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5" Nov 03 20:16:40.135584 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:40.135558 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f\" (UID: \"14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f" Nov 03 20:16:40.148968 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:40.148937 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmz9c\" (UniqueName: \"kubernetes.io/projected/14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3-kube-api-access-pmz9c\") pod \"payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f\" (UID: \"14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f" Nov 03 20:16:40.240250 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:40.236404 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw8kl\" (UniqueName: \"kubernetes.io/projected/a98544fd-0619-4632-b931-718a2348d907-kube-api-access-vw8kl\") pod \"payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5\" (UID: \"a98544fd-0619-4632-b931-718a2348d907\") " pod="openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5" Nov 03 20:16:40.240250 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:40.236761 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a98544fd-0619-4632-b931-718a2348d907-plugins-config-volume\") pod \"payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5\" (UID: \"a98544fd-0619-4632-b931-718a2348d907\") " pod="openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5" Nov 03 20:16:40.240250 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:40.237942 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a98544fd-0619-4632-b931-718a2348d907-plugins-config-volume\") pod \"payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5\" (UID: \"a98544fd-0619-4632-b931-718a2348d907\") " pod="openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5" Nov 03 20:16:40.361547 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:40.361504 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f" Nov 03 20:16:40.362893 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:40.362840 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw8kl\" (UniqueName: \"kubernetes.io/projected/a98544fd-0619-4632-b931-718a2348d907-kube-api-access-vw8kl\") pod \"payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5\" (UID: \"a98544fd-0619-4632-b931-718a2348d907\") " pod="openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5" Nov 03 20:16:40.629621 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:40.629582 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5" Nov 03 20:16:40.686852 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:40.686810 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f"] Nov 03 20:16:40.689327 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:16:40.689295 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14fe21d6_b444_4e0f_b5df_f83fcc4ab4e3.slice/crio-311e66c34080c3a9e2bfbbaa5a177ad4313f246654e454258baee35f95e53670 WatchSource:0}: Error finding container 311e66c34080c3a9e2bfbbaa5a177ad4313f246654e454258baee35f95e53670: Status 404 returned error can't find the container with id 311e66c34080c3a9e2bfbbaa5a177ad4313f246654e454258baee35f95e53670 Nov 03 20:16:40.936289 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:40.936259 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5"] Nov 03 20:16:40.937964 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:16:40.937918 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98544fd_0619_4632_b931_718a2348d907.slice/crio-afba126ff7eaebf2fa4b06811b7c4126daa885e54ccd4d6af29d2a7ebdcd17de WatchSource:0}: Error finding container afba126ff7eaebf2fa4b06811b7c4126daa885e54ccd4d6af29d2a7ebdcd17de: Status 404 returned error can't find the container with id afba126ff7eaebf2fa4b06811b7c4126daa885e54ccd4d6af29d2a7ebdcd17de Nov 03 20:16:41.451765 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:41.451729 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5" event={"ID":"a98544fd-0619-4632-b931-718a2348d907","Type":"ContainerStarted","Data":"62f549b2c233e75f696e81709cd6c16eddee9b2dde65f65ec7a47b4ba5a24e0b"} Nov 03 20:16:41.452166 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:41.451787 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5" event={"ID":"a98544fd-0619-4632-b931-718a2348d907","Type":"ContainerStarted","Data":"afba126ff7eaebf2fa4b06811b7c4126daa885e54ccd4d6af29d2a7ebdcd17de"} Nov 03 20:16:41.452166 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:41.451818 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5" Nov 03 20:16:41.453431 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:41.453402 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f" event={"ID":"14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3","Type":"ContainerStarted","Data":"2b0407853c5ef2e7eec5a0e8ab86dbfaa16b50fcade9f05f0e03caa7b00ebdea"} Nov 03 20:16:41.453527 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:41.453448 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f" event={"ID":"14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3","Type":"ContainerStarted","Data":"311e66c34080c3a9e2bfbbaa5a177ad4313f246654e454258baee35f95e53670"} Nov 03 20:16:41.453527 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:41.453469 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f" Nov 03 20:16:41.471037 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:41.470991 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5" podStartSLOduration=2.470978537 podStartE2EDuration="2.470978537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:41.469401559 +0000 UTC m=+1184.401269070" watchObservedRunningTime="2026-11-03 20:16:41.470978537 +0000 UTC m=+1184.402846065" Nov 03 20:16:41.489515 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:41.489465 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f" podStartSLOduration=2.489451949 podStartE2EDuration="2.489451949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:41.486751459 +0000 UTC m=+1184.418618970" watchObservedRunningTime="2026-11-03 20:16:41.489451949 +0000 UTC m=+1184.421319495" Nov 03 20:16:49.431778 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:49.431739 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv" Nov 03 20:16:49.432163 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:49.431849 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d" Nov 03 20:16:52.463061 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:52.463020 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5" Nov 03 20:16:52.464465 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:52.464436 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f" Nov 03 20:16:57.354979 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:57.354936 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:16:57.355942 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:16:57.355923 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:16:58.211288 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:16:58.211252 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:17:18.396371 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:18.390127 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5"] Nov 03 20:17:18.404768 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:18.404735 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f"] Nov 03 20:17:22.113110 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:22.113068 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb"] Nov 03 20:17:22.156201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:22.133058 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" podUID="46b16107-a3b5-46fe-a4be-7fde061ac3da" containerName="istio-proxy" containerID="cri-o://17951a0f014581820e8bfc4f4843c9e54d59052add312ec9061e56d8c5f4d5fc" gracePeriod=30 Nov 03 20:17:22.431259 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:22.431156 4918 patch_prober.go:29] interesting pod/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:17:22.431500 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:22.431472 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" podUID="46b16107-a3b5-46fe-a4be-7fde061ac3da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:17:22.532833 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:22.532768 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:23.240542 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.240506 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5"] Nov 03 20:17:23.246546 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.246524 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.263802 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.260384 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-3e9176-openshift-default-dockercfg-bwxvx\"" Nov 03 20:17:23.263802 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.263563 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5"] Nov 03 20:17:23.388755 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.387067 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-istio-envoy\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.388755 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.387124 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-workload-socket\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.388755 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.387325 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-istio-data\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.388755 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.387387 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af357454-5d01-46c9-a1ef-bf09169443ee-istio-token\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.388755 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.387417 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhw7q\" (UniqueName: \"kubernetes.io/projected/af357454-5d01-46c9-a1ef-bf09169443ee-kube-api-access-qhw7q\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.388755 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.387450 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af357454-5d01-46c9-a1ef-bf09169443ee-istiod-ca-cert\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.388755 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.387513 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-workload-certs\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.388755 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.387743 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af357454-5d01-46c9-a1ef-bf09169443ee-istio-podinfo\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.388755 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.387814 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-credential-socket\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.398564 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.397278 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5" podUID="a98544fd-0619-4632-b931-718a2348d907" containerName="payload-processing" containerID="cri-o://62f549b2c233e75f696e81709cd6c16eddee9b2dde65f65ec7a47b4ba5a24e0b" gracePeriod=25 Nov 03 20:17:23.415372 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.415320 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f" podUID="14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3" containerName="payload-pre-processing" containerID="cri-o://2b0407853c5ef2e7eec5a0e8ab86dbfaa16b50fcade9f05f0e03caa7b00ebdea" gracePeriod=25 Nov 03 20:17:23.552898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.501398 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-istio-data\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.552898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.501485 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af357454-5d01-46c9-a1ef-bf09169443ee-istio-token\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.552898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.501518 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhw7q\" (UniqueName: \"kubernetes.io/projected/af357454-5d01-46c9-a1ef-bf09169443ee-kube-api-access-qhw7q\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.552898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.501552 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af357454-5d01-46c9-a1ef-bf09169443ee-istiod-ca-cert\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.552898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.501604 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-workload-certs\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.552898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.501761 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af357454-5d01-46c9-a1ef-bf09169443ee-istio-podinfo\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.552898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.501809 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-credential-socket\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.552898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.501900 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-istio-envoy\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.552898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.501931 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-workload-socket\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.552898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.502422 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-workload-socket\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.552898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.502676 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-istio-data\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.552898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.503823 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af357454-5d01-46c9-a1ef-bf09169443ee-istiod-ca-cert\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.552898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.504068 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-workload-certs\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.552898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.516691 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-credential-socket\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.552898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.539859 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhw7q\" (UniqueName: \"kubernetes.io/projected/af357454-5d01-46c9-a1ef-bf09169443ee-kube-api-access-qhw7q\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.552898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.546514 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-istio-envoy\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.552898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.552774 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af357454-5d01-46c9-a1ef-bf09169443ee-istio-token\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.561247 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.561162 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af357454-5d01-46c9-a1ef-bf09169443ee-istio-podinfo\") pod \"e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:23.579760 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:23.579492 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:24.049714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.049669 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5"] Nov 03 20:17:24.053579 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:17:24.053524 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf357454_5d01_46c9_a1ef_bf09169443ee.slice/crio-6b161c8c6845f95589681f45731142ba967abf86277a14f8db2e0e160b977b85 WatchSource:0}: Error finding container 6b161c8c6845f95589681f45731142ba967abf86277a14f8db2e0e160b977b85: Status 404 returned error can't find the container with id 6b161c8c6845f95589681f45731142ba967abf86277a14f8db2e0e160b977b85 Nov 03 20:17:24.058965 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.058920 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:17:24.059082 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.059017 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:17:24.059082 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.059068 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:17:24.080980 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.080939 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f" Nov 03 20:17:24.089117 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.083455 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5" Nov 03 20:17:24.230396 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.228061 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3-plugins-config-volume\") pod \"14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3\" (UID: \"14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3\") " Nov 03 20:17:24.230608 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.230468 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3-kube-api-access-pmz9c\" (UniqueName: \"kubernetes.io/projected/14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3-kube-api-access-pmz9c\") pod \"14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3\" (UID: \"14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3\") " Nov 03 20:17:24.230608 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.230540 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a98544fd-0619-4632-b931-718a2348d907-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a98544fd-0619-4632-b931-718a2348d907-plugins-config-volume\") pod \"a98544fd-0619-4632-b931-718a2348d907\" (UID: \"a98544fd-0619-4632-b931-718a2348d907\") " Nov 03 20:17:24.230741 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.230622 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a98544fd-0619-4632-b931-718a2348d907-kube-api-access-vw8kl\" (UniqueName: \"kubernetes.io/projected/a98544fd-0619-4632-b931-718a2348d907-kube-api-access-vw8kl\") pod \"a98544fd-0619-4632-b931-718a2348d907\" (UID: \"a98544fd-0619-4632-b931-718a2348d907\") " Nov 03 20:17:24.232446 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.228550 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3-plugins-config-volume" pod "14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3" (UID: "14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:24.232446 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.232402 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a98544fd-0619-4632-b931-718a2348d907-plugins-config-volume" pod "a98544fd-0619-4632-b931-718a2348d907" (UID: "a98544fd-0619-4632-b931-718a2348d907"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:24.235749 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.235709 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a98544fd-0619-4632-b931-718a2348d907-kube-api-access-vw8kl" pod "a98544fd-0619-4632-b931-718a2348d907" (UID: "a98544fd-0619-4632-b931-718a2348d907"). InnerVolumeSpecName "kube-api-access-vw8kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:24.235878 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.235837 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3-kube-api-access-pmz9c" pod "14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3" (UID: "14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3"). InnerVolumeSpecName "kube-api-access-pmz9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:24.334006 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.333963 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:24.334006 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.334006 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmz9c\" (UniqueName: \"kubernetes.io/projected/14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3-kube-api-access-pmz9c\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:24.334443 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.334024 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a98544fd-0619-4632-b931-718a2348d907-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:24.334443 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.334038 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw8kl\" (UniqueName: \"kubernetes.io/projected/a98544fd-0619-4632-b931-718a2348d907-kube-api-access-vw8kl\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:24.575250 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.575208 4918 generic.go:356] "Generic (PLEG): container finished" podID="14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3" containerID="2b0407853c5ef2e7eec5a0e8ab86dbfaa16b50fcade9f05f0e03caa7b00ebdea" exitCode=0 Nov 03 20:17:24.575444 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.575353 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f" event={"ID":"14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3","Type":"ContainerDied","Data":"2b0407853c5ef2e7eec5a0e8ab86dbfaa16b50fcade9f05f0e03caa7b00ebdea"} Nov 03 20:17:24.575444 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.575404 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f" event={"ID":"14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3","Type":"ContainerDied","Data":"311e66c34080c3a9e2bfbbaa5a177ad4313f246654e454258baee35f95e53670"} Nov 03 20:17:24.575444 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.575437 4918 scope.go:122] "RemoveContainer" containerID="2b0407853c5ef2e7eec5a0e8ab86dbfaa16b50fcade9f05f0e03caa7b00ebdea" Nov 03 20:17:24.576013 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.575611 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f" Nov 03 20:17:24.580436 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.580388 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" event={"ID":"af357454-5d01-46c9-a1ef-bf09169443ee","Type":"ContainerStarted","Data":"20d4b76c2686d47d81d337cd1b4d9edd311a9e9e0c4da64bca25df990eea9a50"} Nov 03 20:17:24.580559 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.580466 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" event={"ID":"af357454-5d01-46c9-a1ef-bf09169443ee","Type":"ContainerStarted","Data":"6b161c8c6845f95589681f45731142ba967abf86277a14f8db2e0e160b977b85"} Nov 03 20:17:24.581575 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.581558 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:24.582527 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.582500 4918 generic.go:356] "Generic (PLEG): container finished" podID="a98544fd-0619-4632-b931-718a2348d907" containerID="62f549b2c233e75f696e81709cd6c16eddee9b2dde65f65ec7a47b4ba5a24e0b" exitCode=0 Nov 03 20:17:24.582638 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.582624 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5" event={"ID":"a98544fd-0619-4632-b931-718a2348d907","Type":"ContainerDied","Data":"62f549b2c233e75f696e81709cd6c16eddee9b2dde65f65ec7a47b4ba5a24e0b"} Nov 03 20:17:24.582740 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.582664 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5" event={"ID":"a98544fd-0619-4632-b931-718a2348d907","Type":"ContainerDied","Data":"afba126ff7eaebf2fa4b06811b7c4126daa885e54ccd4d6af29d2a7ebdcd17de"} Nov 03 20:17:24.582808 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.582738 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5" Nov 03 20:17:24.599582 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.599560 4918 scope.go:122] "RemoveContainer" containerID="2b0407853c5ef2e7eec5a0e8ab86dbfaa16b50fcade9f05f0e03caa7b00ebdea" Nov 03 20:17:24.600058 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:17:24.600031 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b0407853c5ef2e7eec5a0e8ab86dbfaa16b50fcade9f05f0e03caa7b00ebdea\": container with ID starting with 2b0407853c5ef2e7eec5a0e8ab86dbfaa16b50fcade9f05f0e03caa7b00ebdea not found: ID does not exist" containerID="2b0407853c5ef2e7eec5a0e8ab86dbfaa16b50fcade9f05f0e03caa7b00ebdea" Nov 03 20:17:24.600218 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.600072 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0407853c5ef2e7eec5a0e8ab86dbfaa16b50fcade9f05f0e03caa7b00ebdea"} err="failed to get container status \"2b0407853c5ef2e7eec5a0e8ab86dbfaa16b50fcade9f05f0e03caa7b00ebdea\": rpc error: code = NotFound desc = could not find container \"2b0407853c5ef2e7eec5a0e8ab86dbfaa16b50fcade9f05f0e03caa7b00ebdea\": container with ID starting with 2b0407853c5ef2e7eec5a0e8ab86dbfaa16b50fcade9f05f0e03caa7b00ebdea not found: ID does not exist" Nov 03 20:17:24.600218 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.600097 4918 scope.go:122] "RemoveContainer" containerID="62f549b2c233e75f696e81709cd6c16eddee9b2dde65f65ec7a47b4ba5a24e0b" Nov 03 20:17:24.619481 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.619121 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" podStartSLOduration=1.61910212 podStartE2EDuration="1.61910212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:24.616565204 +0000 UTC m=+1227.548432715" watchObservedRunningTime="2026-11-03 20:17:24.61910212 +0000 UTC m=+1227.550969635" Nov 03 20:17:24.620109 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.619740 4918 scope.go:122] "RemoveContainer" containerID="62f549b2c233e75f696e81709cd6c16eddee9b2dde65f65ec7a47b4ba5a24e0b" Nov 03 20:17:24.620645 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:17:24.620498 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f549b2c233e75f696e81709cd6c16eddee9b2dde65f65ec7a47b4ba5a24e0b\": container with ID starting with 62f549b2c233e75f696e81709cd6c16eddee9b2dde65f65ec7a47b4ba5a24e0b not found: ID does not exist" containerID="62f549b2c233e75f696e81709cd6c16eddee9b2dde65f65ec7a47b4ba5a24e0b" Nov 03 20:17:24.620645 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.620534 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f549b2c233e75f696e81709cd6c16eddee9b2dde65f65ec7a47b4ba5a24e0b"} err="failed to get container status \"62f549b2c233e75f696e81709cd6c16eddee9b2dde65f65ec7a47b4ba5a24e0b\": rpc error: code = NotFound desc = could not find container \"62f549b2c233e75f696e81709cd6c16eddee9b2dde65f65ec7a47b4ba5a24e0b\": container with ID starting with 62f549b2c233e75f696e81709cd6c16eddee9b2dde65f65ec7a47b4ba5a24e0b not found: ID does not exist" Nov 03 20:17:24.636359 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.636324 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f"] Nov 03 20:17:24.650331 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.650293 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-11b0d340-677cbfc6c-z659f"] Nov 03 20:17:24.663906 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.663731 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5"] Nov 03 20:17:24.670617 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:24.670584 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-11b0d340-7f894d78f4-sjhh5"] Nov 03 20:17:25.217654 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:25.217613 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3" path="/var/lib/kubelet/pods/14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3/volumes" Nov 03 20:17:25.218208 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:25.218170 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a98544fd-0619-4632-b931-718a2348d907" path="/var/lib/kubelet/pods/a98544fd-0619-4632-b931-718a2348d907/volumes" Nov 03 20:17:25.588918 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:25.588652 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:25.596655 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:25.596515 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:25.597357 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:25.597336 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:17:28.133711 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.133677 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7"] Nov 03 20:17:28.134430 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.134412 4918 cpu_manager.go:414] "containerMap: removing container" podUID="14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3" containerName="payload-pre-processing" Nov 03 20:17:28.134491 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.134430 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3" containerName="payload-pre-processing" Nov 03 20:17:28.134491 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.134444 4918 cpu_manager.go:414] "containerMap: removing container" podUID="a98544fd-0619-4632-b931-718a2348d907" containerName="payload-processing" Nov 03 20:17:28.134491 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.134450 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a98544fd-0619-4632-b931-718a2348d907" containerName="payload-processing" Nov 03 20:17:28.134595 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.134548 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="14fe21d6-b444-4e0f-b5df-f83fcc4ab4e3" containerName="payload-pre-processing" Nov 03 20:17:28.134595 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.134556 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a98544fd-0619-4632-b931-718a2348d907" containerName="payload-processing" Nov 03 20:17:28.154246 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.154211 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7"] Nov 03 20:17:28.154246 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.154231 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7" Nov 03 20:17:28.158285 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.158259 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-3e9176\"" Nov 03 20:17:28.159594 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.159561 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-3e9176-dockercfg-vxcf6\"" Nov 03 20:17:28.185344 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.185317 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m"] Nov 03 20:17:28.201590 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.201559 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m"] Nov 03 20:17:28.201747 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.201722 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m" Nov 03 20:17:28.226075 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:17:28.225899 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:17:28.239105 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.228891 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5ms2\" (UniqueName: \"kubernetes.io/projected/4e6db5de-2704-45a9-8e69-a78d3cf1fb5a-kube-api-access-g5ms2\") pod \"payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7\" (UID: \"4e6db5de-2704-45a9-8e69-a78d3cf1fb5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7" Nov 03 20:17:28.239105 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.229107 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6a964b-7ff6-4c06-b46f-499c03cfd744-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m\" (UID: \"1a6a964b-7ff6-4c06-b46f-499c03cfd744\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m" Nov 03 20:17:28.239105 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.229162 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr24j\" (UniqueName: \"kubernetes.io/projected/1a6a964b-7ff6-4c06-b46f-499c03cfd744-kube-api-access-kr24j\") pod \"payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m\" (UID: \"1a6a964b-7ff6-4c06-b46f-499c03cfd744\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m" Nov 03 20:17:28.239105 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.229356 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e6db5de-2704-45a9-8e69-a78d3cf1fb5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7\" (UID: \"4e6db5de-2704-45a9-8e69-a78d3cf1fb5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7" Nov 03 20:17:28.342207 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.333226 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e6db5de-2704-45a9-8e69-a78d3cf1fb5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7\" (UID: \"4e6db5de-2704-45a9-8e69-a78d3cf1fb5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7" Nov 03 20:17:28.387236 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.345988 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e6db5de-2704-45a9-8e69-a78d3cf1fb5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7\" (UID: \"4e6db5de-2704-45a9-8e69-a78d3cf1fb5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7" Nov 03 20:17:28.387236 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.379137 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5ms2\" (UniqueName: \"kubernetes.io/projected/4e6db5de-2704-45a9-8e69-a78d3cf1fb5a-kube-api-access-g5ms2\") pod \"payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7\" (UID: \"4e6db5de-2704-45a9-8e69-a78d3cf1fb5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7" Nov 03 20:17:28.387236 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.379374 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6a964b-7ff6-4c06-b46f-499c03cfd744-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m\" (UID: \"1a6a964b-7ff6-4c06-b46f-499c03cfd744\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m" Nov 03 20:17:28.387236 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.379438 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr24j\" (UniqueName: \"kubernetes.io/projected/1a6a964b-7ff6-4c06-b46f-499c03cfd744-kube-api-access-kr24j\") pod \"payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m\" (UID: \"1a6a964b-7ff6-4c06-b46f-499c03cfd744\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m" Nov 03 20:17:28.387236 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.382523 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6a964b-7ff6-4c06-b46f-499c03cfd744-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m\" (UID: \"1a6a964b-7ff6-4c06-b46f-499c03cfd744\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m" Nov 03 20:17:28.426220 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.426031 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5ms2\" (UniqueName: \"kubernetes.io/projected/4e6db5de-2704-45a9-8e69-a78d3cf1fb5a-kube-api-access-g5ms2\") pod \"payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7\" (UID: \"4e6db5de-2704-45a9-8e69-a78d3cf1fb5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7" Nov 03 20:17:28.446393 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.442357 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr24j\" (UniqueName: \"kubernetes.io/projected/1a6a964b-7ff6-4c06-b46f-499c03cfd744-kube-api-access-kr24j\") pod \"payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m\" (UID: \"1a6a964b-7ff6-4c06-b46f-499c03cfd744\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m" Nov 03 20:17:28.485450 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.485413 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7" Nov 03 20:17:28.537918 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.537886 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m" Nov 03 20:17:28.695106 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.695058 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" event={"ID":"46b16107-a3b5-46fe-a4be-7fde061ac3da","Type":"ContainerDied","Data":"17951a0f014581820e8bfc4f4843c9e54d59052add312ec9061e56d8c5f4d5fc"} Nov 03 20:17:28.695417 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.695120 4918 generic.go:356] "Generic (PLEG): container finished" podID="46b16107-a3b5-46fe-a4be-7fde061ac3da" containerID="17951a0f014581820e8bfc4f4843c9e54d59052add312ec9061e56d8c5f4d5fc" exitCode=0 Nov 03 20:17:28.699032 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.699002 4918 generic.go:356] "Generic (PLEG): container finished" podID="c878724e-bf1c-4ff5-836b-af4e0304c6dc" containerID="e9a1d2237222b859f6f800faca16bb42f9b604e85ac6ea529af4dea003228638" exitCode=0 Nov 03 20:17:28.700356 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.699166 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" event={"ID":"c878724e-bf1c-4ff5-836b-af4e0304c6dc","Type":"ContainerDied","Data":"e9a1d2237222b859f6f800faca16bb42f9b604e85ac6ea529af4dea003228638"} Nov 03 20:17:28.700356 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.700196 4918 scope.go:122] "RemoveContainer" containerID="e9a1d2237222b859f6f800faca16bb42f9b604e85ac6ea529af4dea003228638" Nov 03 20:17:28.702469 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.702436 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:17:28.702671 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.702648 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:17:28.702828 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.702798 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:17:28.847637 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.847303 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m"] Nov 03 20:17:28.890685 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:17:28.888610 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6db5de_2704_45a9_8e69_a78d3cf1fb5a.slice/crio-d87dcc2f4090169ccee44353bf93e4cca79ea68429cd695b567569019368adc3 WatchSource:0}: Error finding container d87dcc2f4090169ccee44353bf93e4cca79ea68429cd695b567569019368adc3: Status 404 returned error can't find the container with id d87dcc2f4090169ccee44353bf93e4cca79ea68429cd695b567569019368adc3 Nov 03 20:17:28.894269 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:28.894245 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7"] Nov 03 20:17:29.004358 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.004333 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:17:29.204234 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.204086 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-data\") pod \"46b16107-a3b5-46fe-a4be-7fde061ac3da\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " Nov 03 20:17:29.204234 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.204166 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-envoy\") pod \"46b16107-a3b5-46fe-a4be-7fde061ac3da\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " Nov 03 20:17:29.204774 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.204349 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-podinfo\") pod \"46b16107-a3b5-46fe-a4be-7fde061ac3da\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " Nov 03 20:17:29.204774 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.204404 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-credential-socket\") pod \"46b16107-a3b5-46fe-a4be-7fde061ac3da\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " Nov 03 20:17:29.204774 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.204464 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-workload-socket\") pod \"46b16107-a3b5-46fe-a4be-7fde061ac3da\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " Nov 03 20:17:29.204774 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.204533 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b16107-a3b5-46fe-a4be-7fde061ac3da-kube-api-access-f7tkt\" (UniqueName: \"kubernetes.io/projected/46b16107-a3b5-46fe-a4be-7fde061ac3da-kube-api-access-f7tkt\") pod \"46b16107-a3b5-46fe-a4be-7fde061ac3da\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " Nov 03 20:17:29.204774 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.204566 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-workload-certs\") pod \"46b16107-a3b5-46fe-a4be-7fde061ac3da\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " Nov 03 20:17:29.204774 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.204566 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-data" pod "46b16107-a3b5-46fe-a4be-7fde061ac3da" (UID: "46b16107-a3b5-46fe-a4be-7fde061ac3da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:29.204774 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.204682 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b16107-a3b5-46fe-a4be-7fde061ac3da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b16107-a3b5-46fe-a4be-7fde061ac3da-istiod-ca-cert\") pod \"46b16107-a3b5-46fe-a4be-7fde061ac3da\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " Nov 03 20:17:29.204774 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.204712 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-token\" (UniqueName: \"kubernetes.io/projected/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-token\") pod \"46b16107-a3b5-46fe-a4be-7fde061ac3da\" (UID: \"46b16107-a3b5-46fe-a4be-7fde061ac3da\") " Nov 03 20:17:29.205209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.204811 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-workload-certs" pod "46b16107-a3b5-46fe-a4be-7fde061ac3da" (UID: "46b16107-a3b5-46fe-a4be-7fde061ac3da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:29.205363 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.205337 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-credential-socket" pod "46b16107-a3b5-46fe-a4be-7fde061ac3da" (UID: "46b16107-a3b5-46fe-a4be-7fde061ac3da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:29.205599 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.205574 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:29.205679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.205599 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:29.205849 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.205828 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b16107-a3b5-46fe-a4be-7fde061ac3da-istiod-ca-cert" pod "46b16107-a3b5-46fe-a4be-7fde061ac3da" (UID: "46b16107-a3b5-46fe-a4be-7fde061ac3da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:29.206687 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.206664 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-workload-socket" pod "46b16107-a3b5-46fe-a4be-7fde061ac3da" (UID: "46b16107-a3b5-46fe-a4be-7fde061ac3da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:29.207368 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.207340 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-envoy" pod "46b16107-a3b5-46fe-a4be-7fde061ac3da" (UID: "46b16107-a3b5-46fe-a4be-7fde061ac3da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:29.207967 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.207910 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-podinfo" pod "46b16107-a3b5-46fe-a4be-7fde061ac3da" (UID: "46b16107-a3b5-46fe-a4be-7fde061ac3da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:17:29.208603 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.208579 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-token" pod "46b16107-a3b5-46fe-a4be-7fde061ac3da" (UID: "46b16107-a3b5-46fe-a4be-7fde061ac3da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:29.210245 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.210221 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b16107-a3b5-46fe-a4be-7fde061ac3da-kube-api-access-f7tkt" pod "46b16107-a3b5-46fe-a4be-7fde061ac3da" (UID: "46b16107-a3b5-46fe-a4be-7fde061ac3da"). InnerVolumeSpecName "kube-api-access-f7tkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:29.307329 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.307297 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:29.307329 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.307327 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:29.307329 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.307337 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:29.307593 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.307346 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7tkt\" (UniqueName: \"kubernetes.io/projected/46b16107-a3b5-46fe-a4be-7fde061ac3da-kube-api-access-f7tkt\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:29.307593 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.307357 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b16107-a3b5-46fe-a4be-7fde061ac3da-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:29.307593 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.307366 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:29.307593 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.307374 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b16107-a3b5-46fe-a4be-7fde061ac3da-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:29.715210 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.715109 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" event={"ID":"46b16107-a3b5-46fe-a4be-7fde061ac3da","Type":"ContainerDied","Data":"8ae0a8b7b8161d50fbf6bf6796c00c917c8c6620700c3037f76d8fa5744e2db8"} Nov 03 20:17:29.715514 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.715478 4918 scope.go:122] "RemoveContainer" containerID="17951a0f014581820e8bfc4f4843c9e54d59052add312ec9061e56d8c5f4d5fc" Nov 03 20:17:29.715851 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.715813 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb" Nov 03 20:17:29.744601 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.744567 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" event={"ID":"c878724e-bf1c-4ff5-836b-af4e0304c6dc","Type":"ContainerStarted","Data":"6189419cd87e5bbe6ab9d82da44147773b6f5e09cc95422f860d9ab3d753839d"} Nov 03 20:17:29.748749 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.748718 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m" event={"ID":"1a6a964b-7ff6-4c06-b46f-499c03cfd744","Type":"ContainerStarted","Data":"38276d246d6865295bd91b2dc9479a1bb8d96dfc418956e60005919a2b69ef25"} Nov 03 20:17:29.749844 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.749807 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m" Nov 03 20:17:29.756251 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.756218 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m" event={"ID":"1a6a964b-7ff6-4c06-b46f-499c03cfd744","Type":"ContainerStarted","Data":"dba7759d3695d994cd51acd86e36c639e839cf361ff8e1cb4919fdec0529e63e"} Nov 03 20:17:29.759646 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.759618 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7" event={"ID":"4e6db5de-2704-45a9-8e69-a78d3cf1fb5a","Type":"ContainerStarted","Data":"fc5023a86df4184899628bba8bc97c4d89b28f11f6ed56d8394eb1cf990d9b12"} Nov 03 20:17:29.759904 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.759819 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7" event={"ID":"4e6db5de-2704-45a9-8e69-a78d3cf1fb5a","Type":"ContainerStarted","Data":"d87dcc2f4090169ccee44353bf93e4cca79ea68429cd695b567569019368adc3"} Nov 03 20:17:29.760497 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.760434 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb"] Nov 03 20:17:29.766009 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.765988 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:17:29.766691 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.766637 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7" Nov 03 20:17:29.778260 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.775996 4918 patch_prober.go:29] interesting pod/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:17:29.778260 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.776061 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" podUID="c878724e-bf1c-4ff5-836b-af4e0304c6dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 20:17:29.849335 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.849303 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-11b0d340-openshift-default-549b69f485-fnbhb"] Nov 03 20:17:29.854564 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.853398 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m" podStartSLOduration=1.853379464 podStartE2EDuration="1.853379464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:29.788406882 +0000 UTC m=+1232.720274393" watchObservedRunningTime="2026-11-03 20:17:29.853379464 +0000 UTC m=+1232.785246978" Nov 03 20:17:29.875839 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:29.875789 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7" podStartSLOduration=1.875774998 podStartE2EDuration="1.875774998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:29.866256314 +0000 UTC m=+1232.798123825" watchObservedRunningTime="2026-11-03 20:17:29.875774998 +0000 UTC m=+1232.807642527" Nov 03 20:17:30.674027 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:30.673996 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv"] Nov 03 20:17:30.681310 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:30.681280 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d"] Nov 03 20:17:30.761787 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:30.761753 4918 patch_prober.go:29] interesting pod/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:17:30.762040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:30.762009 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" podUID="c878724e-bf1c-4ff5-836b-af4e0304c6dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 20:17:31.216996 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:31.216955 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b16107-a3b5-46fe-a4be-7fde061ac3da" path="/var/lib/kubelet/pods/46b16107-a3b5-46fe-a4be-7fde061ac3da/volumes" Nov 03 20:17:31.761943 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:31.761911 4918 patch_prober.go:29] interesting pod/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:17:31.762367 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:31.761962 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" podUID="c878724e-bf1c-4ff5-836b-af4e0304c6dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 20:17:32.761406 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:32.761372 4918 patch_prober.go:29] interesting pod/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:17:32.761580 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:32.761436 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" podUID="c878724e-bf1c-4ff5-836b-af4e0304c6dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 20:17:33.449926 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.449883 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8"] Nov 03 20:17:33.451011 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.450952 4918 cpu_manager.go:414] "containerMap: removing container" podUID="46b16107-a3b5-46fe-a4be-7fde061ac3da" containerName="istio-proxy" Nov 03 20:17:33.451011 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.450976 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b16107-a3b5-46fe-a4be-7fde061ac3da" containerName="istio-proxy" Nov 03 20:17:33.451317 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.451156 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b16107-a3b5-46fe-a4be-7fde061ac3da" containerName="istio-proxy" Nov 03 20:17:33.457443 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.456708 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" Nov 03 20:17:33.458846 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.458820 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8"] Nov 03 20:17:33.461120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.461088 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-3e9176-serving-cert\"" Nov 03 20:17:33.557210 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.553474 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lzrg\" (UniqueName: \"kubernetes.io/projected/4c77123e-9c94-4299-a7e0-1759315d24fd-kube-api-access-9lzrg\") pod \"maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8\" (UID: \"4c77123e-9c94-4299-a7e0-1759315d24fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" Nov 03 20:17:33.557761 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.557729 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c77123e-9c94-4299-a7e0-1759315d24fd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8\" (UID: \"4c77123e-9c94-4299-a7e0-1759315d24fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" Nov 03 20:17:33.558307 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.558247 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c77123e-9c94-4299-a7e0-1759315d24fd-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8\" (UID: \"4c77123e-9c94-4299-a7e0-1759315d24fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" Nov 03 20:17:33.568234 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.565220 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c77123e-9c94-4299-a7e0-1759315d24fd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8\" (UID: \"4c77123e-9c94-4299-a7e0-1759315d24fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" Nov 03 20:17:33.670064 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.670030 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c77123e-9c94-4299-a7e0-1759315d24fd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8\" (UID: \"4c77123e-9c94-4299-a7e0-1759315d24fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" Nov 03 20:17:33.670298 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.670089 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c77123e-9c94-4299-a7e0-1759315d24fd-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8\" (UID: \"4c77123e-9c94-4299-a7e0-1759315d24fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" Nov 03 20:17:33.670298 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.670167 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c77123e-9c94-4299-a7e0-1759315d24fd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8\" (UID: \"4c77123e-9c94-4299-a7e0-1759315d24fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" Nov 03 20:17:33.670423 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.670358 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lzrg\" (UniqueName: \"kubernetes.io/projected/4c77123e-9c94-4299-a7e0-1759315d24fd-kube-api-access-9lzrg\") pod \"maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8\" (UID: \"4c77123e-9c94-4299-a7e0-1759315d24fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" Nov 03 20:17:33.670675 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.670645 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c77123e-9c94-4299-a7e0-1759315d24fd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8\" (UID: \"4c77123e-9c94-4299-a7e0-1759315d24fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" Nov 03 20:17:33.672633 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.672604 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c77123e-9c94-4299-a7e0-1759315d24fd-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8\" (UID: \"4c77123e-9c94-4299-a7e0-1759315d24fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" Nov 03 20:17:33.672737 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.672711 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c77123e-9c94-4299-a7e0-1759315d24fd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8\" (UID: \"4c77123e-9c94-4299-a7e0-1759315d24fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" Nov 03 20:17:33.684946 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.684909 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lzrg\" (UniqueName: \"kubernetes.io/projected/4c77123e-9c94-4299-a7e0-1759315d24fd-kube-api-access-9lzrg\") pod \"maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8\" (UID: \"4c77123e-9c94-4299-a7e0-1759315d24fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" Nov 03 20:17:33.761236 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.761141 4918 patch_prober.go:29] interesting pod/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:17:33.761372 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.761226 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" podUID="c878724e-bf1c-4ff5-836b-af4e0304c6dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 20:17:33.780778 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:33.780645 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" Nov 03 20:17:34.282778 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:34.282744 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:17:34.360720 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:34.360698 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8"] Nov 03 20:17:34.761901 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:34.761544 4918 patch_prober.go:29] interesting pod/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:17:34.761901 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:34.761611 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" podUID="c878724e-bf1c-4ff5-836b-af4e0304c6dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 20:17:34.812748 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:34.812692 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" event={"ID":"4c77123e-9c94-4299-a7e0-1759315d24fd","Type":"ContainerStarted","Data":"165712305d6bc46bd4d709d7c1be005da969127a04c7ff8e2136994c552db69b"} Nov 03 20:17:35.675584 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:35.675540 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv" podUID="8f0392da-1544-488e-a52b-b0a3508b3dc6" containerName="payload-processing" containerID="cri-o://9b672e6c562882545a91c98d4237336e8e6b8ebadefcce80ff2e4798c97b4f58" gracePeriod=25 Nov 03 20:17:35.682151 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:35.682094 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d" podUID="84e9bbf6-e6cd-47c2-9761-d5b0aff056f0" containerName="payload-pre-processing" containerID="cri-o://95ca591a606d8c6e69ad3819f0c02c80868251f17953682243962efbb776b9e4" gracePeriod=25 Nov 03 20:17:35.781104 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:35.780936 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:17:35.785662 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:35.785636 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:17:35.824090 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:35.824045 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" event={"ID":"4c77123e-9c94-4299-a7e0-1759315d24fd","Type":"ContainerStarted","Data":"699b538d19fb2fa483c35e211d4b92d0d8aa2ff2a0440402a5eb25c6ee39962e"} Nov 03 20:17:35.825049 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:35.825027 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" Nov 03 20:17:35.827321 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:35.827287 4918 generic.go:356] "Generic (PLEG): container finished" podID="8f0392da-1544-488e-a52b-b0a3508b3dc6" containerID="9b672e6c562882545a91c98d4237336e8e6b8ebadefcce80ff2e4798c97b4f58" exitCode=0 Nov 03 20:17:35.827446 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:35.827430 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv" event={"ID":"8f0392da-1544-488e-a52b-b0a3508b3dc6","Type":"ContainerDied","Data":"9b672e6c562882545a91c98d4237336e8e6b8ebadefcce80ff2e4798c97b4f58"} Nov 03 20:17:35.830377 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:35.830270 4918 generic.go:356] "Generic (PLEG): container finished" podID="84e9bbf6-e6cd-47c2-9761-d5b0aff056f0" containerID="95ca591a606d8c6e69ad3819f0c02c80868251f17953682243962efbb776b9e4" exitCode=0 Nov 03 20:17:35.831495 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:35.831347 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d" event={"ID":"84e9bbf6-e6cd-47c2-9761-d5b0aff056f0","Type":"ContainerDied","Data":"95ca591a606d8c6e69ad3819f0c02c80868251f17953682243962efbb776b9e4"} Nov 03 20:17:35.868220 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:35.858149 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" podStartSLOduration=2.330329402 podStartE2EDuration="2.858130238s" totalImagesPullingTime="527.800836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:35.855640303 +0000 UTC m=+1238.787507815" watchObservedRunningTime="2026-11-03 20:17:35.858130238 +0000 UTC m=+1238.789997750" Nov 03 20:17:36.159431 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.157829 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv" Nov 03 20:17:36.162469 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.162408 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d" Nov 03 20:17:36.264059 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.261835 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f0392da-1544-488e-a52b-b0a3508b3dc6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0392da-1544-488e-a52b-b0a3508b3dc6-plugins-config-volume\") pod \"8f0392da-1544-488e-a52b-b0a3508b3dc6\" (UID: \"8f0392da-1544-488e-a52b-b0a3508b3dc6\") " Nov 03 20:17:36.264059 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.261906 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e9bbf6-e6cd-47c2-9761-d5b0aff056f0-kube-api-access-7c5vb\" (UniqueName: \"kubernetes.io/projected/84e9bbf6-e6cd-47c2-9761-d5b0aff056f0-kube-api-access-7c5vb\") pod \"84e9bbf6-e6cd-47c2-9761-d5b0aff056f0\" (UID: \"84e9bbf6-e6cd-47c2-9761-d5b0aff056f0\") " Nov 03 20:17:36.264059 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.262030 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0392da-1544-488e-a52b-b0a3508b3dc6-kube-api-access-qj85g\" (UniqueName: \"kubernetes.io/projected/8f0392da-1544-488e-a52b-b0a3508b3dc6-kube-api-access-qj85g\") pod \"8f0392da-1544-488e-a52b-b0a3508b3dc6\" (UID: \"8f0392da-1544-488e-a52b-b0a3508b3dc6\") " Nov 03 20:17:36.264059 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.262217 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e9bbf6-e6cd-47c2-9761-d5b0aff056f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e9bbf6-e6cd-47c2-9761-d5b0aff056f0-plugins-config-volume\") pod \"84e9bbf6-e6cd-47c2-9761-d5b0aff056f0\" (UID: \"84e9bbf6-e6cd-47c2-9761-d5b0aff056f0\") " Nov 03 20:17:36.264059 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.263346 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e9bbf6-e6cd-47c2-9761-d5b0aff056f0-plugins-config-volume" pod "84e9bbf6-e6cd-47c2-9761-d5b0aff056f0" (UID: "84e9bbf6-e6cd-47c2-9761-d5b0aff056f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:36.264059 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.263739 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0392da-1544-488e-a52b-b0a3508b3dc6-plugins-config-volume" pod "8f0392da-1544-488e-a52b-b0a3508b3dc6" (UID: "8f0392da-1544-488e-a52b-b0a3508b3dc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:36.268066 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.268037 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0392da-1544-488e-a52b-b0a3508b3dc6-kube-api-access-qj85g" pod "8f0392da-1544-488e-a52b-b0a3508b3dc6" (UID: "8f0392da-1544-488e-a52b-b0a3508b3dc6"). InnerVolumeSpecName "kube-api-access-qj85g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:36.271312 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.271286 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e9bbf6-e6cd-47c2-9761-d5b0aff056f0-kube-api-access-7c5vb" pod "84e9bbf6-e6cd-47c2-9761-d5b0aff056f0" (UID: "84e9bbf6-e6cd-47c2-9761-d5b0aff056f0"). InnerVolumeSpecName "kube-api-access-7c5vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:36.372242 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.365654 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj85g\" (UniqueName: \"kubernetes.io/projected/8f0392da-1544-488e-a52b-b0a3508b3dc6-kube-api-access-qj85g\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:36.372242 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.365708 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e9bbf6-e6cd-47c2-9761-d5b0aff056f0-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:36.372242 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.365733 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0392da-1544-488e-a52b-b0a3508b3dc6-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:36.372242 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.365758 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c5vb\" (UniqueName: \"kubernetes.io/projected/84e9bbf6-e6cd-47c2-9761-d5b0aff056f0-kube-api-access-7c5vb\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:36.838805 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.838509 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv" Nov 03 20:17:36.838805 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.838541 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv" event={"ID":"8f0392da-1544-488e-a52b-b0a3508b3dc6","Type":"ContainerDied","Data":"0813000b747bbfef6e8c4ed905494bf3a45d582e6c9838109f2595c947048cde"} Nov 03 20:17:36.838805 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.838610 4918 scope.go:122] "RemoveContainer" containerID="9b672e6c562882545a91c98d4237336e8e6b8ebadefcce80ff2e4798c97b4f58" Nov 03 20:17:36.840548 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.840525 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d" Nov 03 20:17:36.840680 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.840528 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d" event={"ID":"84e9bbf6-e6cd-47c2-9761-d5b0aff056f0","Type":"ContainerDied","Data":"f9d98cda9f5ccc8c03184f6679b0a54dc8aeb22374c42fcdeaec4c07da170864"} Nov 03 20:17:36.857406 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.857388 4918 scope.go:122] "RemoveContainer" containerID="95ca591a606d8c6e69ad3819f0c02c80868251f17953682243962efbb776b9e4" Nov 03 20:17:36.874495 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.874472 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d"] Nov 03 20:17:36.878974 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.878953 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ffd1cf5a-79776f4595-dv92d"] Nov 03 20:17:36.890283 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.890260 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv"] Nov 03 20:17:36.894172 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:36.894141 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ffd1cf5a-7ff78f468b-kwnnv"] Nov 03 20:17:37.223719 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:37.223640 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e9bbf6-e6cd-47c2-9761-d5b0aff056f0" path="/var/lib/kubelet/pods/84e9bbf6-e6cd-47c2-9761-d5b0aff056f0/volumes" Nov 03 20:17:37.224091 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:37.224075 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f0392da-1544-488e-a52b-b0a3508b3dc6" path="/var/lib/kubelet/pods/8f0392da-1544-488e-a52b-b0a3508b3dc6/volumes" Nov 03 20:17:37.563514 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:37.563482 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf"] Nov 03 20:17:37.563764 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:37.563733 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" podUID="c878724e-bf1c-4ff5-836b-af4e0304c6dc" containerName="istio-proxy" containerID="cri-o://6189419cd87e5bbe6ab9d82da44147773b6f5e09cc95422f860d9ab3d753839d" gracePeriod=30 Nov 03 20:17:38.830925 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:38.830893 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn"] Nov 03 20:17:38.831598 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:38.831581 4918 cpu_manager.go:414] "containerMap: removing container" podUID="8f0392da-1544-488e-a52b-b0a3508b3dc6" containerName="payload-processing" Nov 03 20:17:38.831598 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:38.831599 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0392da-1544-488e-a52b-b0a3508b3dc6" containerName="payload-processing" Nov 03 20:17:38.831695 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:38.831660 4918 cpu_manager.go:414] "containerMap: removing container" podUID="84e9bbf6-e6cd-47c2-9761-d5b0aff056f0" containerName="payload-pre-processing" Nov 03 20:17:38.831695 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:38.831670 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e9bbf6-e6cd-47c2-9761-d5b0aff056f0" containerName="payload-pre-processing" Nov 03 20:17:38.831806 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:38.831795 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e9bbf6-e6cd-47c2-9761-d5b0aff056f0" containerName="payload-pre-processing" Nov 03 20:17:38.831848 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:38.831810 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0392da-1544-488e-a52b-b0a3508b3dc6" containerName="payload-processing" Nov 03 20:17:38.837138 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:38.837090 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:38.842656 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:38.842560 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-b35c30-openshift-default-dockercfg-wr7h7\"" Nov 03 20:17:38.854564 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:38.853279 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn"] Nov 03 20:17:38.934882 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:38.934848 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-workload-certs\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:38.935047 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:38.934912 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s72xw\" (UniqueName: \"kubernetes.io/projected/9fb1ff44-4b90-47da-a252-d5771f160330-kube-api-access-s72xw\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:38.935137 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:38.935044 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-istio-data\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:38.935137 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:38.935108 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fb1ff44-4b90-47da-a252-d5771f160330-istio-token\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:38.935255 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:38.935158 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fb1ff44-4b90-47da-a252-d5771f160330-istiod-ca-cert\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:38.935313 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:38.935295 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-credential-socket\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:38.935355 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:38.935329 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-istio-envoy\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:38.935394 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:38.935388 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-workload-socket\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:38.935497 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:38.935485 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fb1ff44-4b90-47da-a252-d5771f160330-istio-podinfo\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:39.037153 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.037120 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-credential-socket\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:39.037335 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.037164 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-istio-envoy\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:39.037335 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.037249 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-workload-socket\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:39.037477 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.037339 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fb1ff44-4b90-47da-a252-d5771f160330-istio-podinfo\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:39.037477 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.037394 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-workload-certs\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:39.037595 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.037563 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s72xw\" (UniqueName: \"kubernetes.io/projected/9fb1ff44-4b90-47da-a252-d5771f160330-kube-api-access-s72xw\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:39.037652 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.037612 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-credential-socket\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:39.037708 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.037684 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-workload-socket\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:39.037763 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.037687 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-istio-data\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:39.037763 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.037725 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-workload-certs\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:39.037881 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.037777 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fb1ff44-4b90-47da-a252-d5771f160330-istio-token\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:39.037881 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.037813 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fb1ff44-4b90-47da-a252-d5771f160330-istiod-ca-cert\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:39.037981 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.037913 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-istio-data\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:39.038492 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.038464 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fb1ff44-4b90-47da-a252-d5771f160330-istiod-ca-cert\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:39.039605 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.039584 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-istio-envoy\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:39.039958 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.039938 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fb1ff44-4b90-47da-a252-d5771f160330-istio-podinfo\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:39.049302 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.049282 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fb1ff44-4b90-47da-a252-d5771f160330-istio-token\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:39.049476 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.049459 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s72xw\" (UniqueName: \"kubernetes.io/projected/9fb1ff44-4b90-47da-a252-d5771f160330-kube-api-access-s72xw\") pod \"e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:39.165256 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.164567 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:39.792706 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.792676 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn"] Nov 03 20:17:39.793309 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:17:39.793283 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb1ff44_4b90_47da_a252_d5771f160330.slice/crio-e532b728c34063c75374c40c512fefe61df0f52e47eefdec9df79a1687501f34 WatchSource:0}: Error finding container e532b728c34063c75374c40c512fefe61df0f52e47eefdec9df79a1687501f34: Status 404 returned error can't find the container with id e532b728c34063c75374c40c512fefe61df0f52e47eefdec9df79a1687501f34 Nov 03 20:17:39.796047 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.796015 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:17:39.796130 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.796084 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:17:39.796130 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.796113 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:17:39.885625 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:39.885583 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" event={"ID":"9fb1ff44-4b90-47da-a252-d5771f160330","Type":"ContainerStarted","Data":"e532b728c34063c75374c40c512fefe61df0f52e47eefdec9df79a1687501f34"} Nov 03 20:17:40.778482 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:40.778454 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m" Nov 03 20:17:40.893667 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:40.893627 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" event={"ID":"9fb1ff44-4b90-47da-a252-d5771f160330","Type":"ContainerStarted","Data":"82bc338730c7f8bd7bd15db32eb5bf81f54b601a7e1da65350bcef80e6b9a38f"} Nov 03 20:17:40.929215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:40.929149 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" podStartSLOduration=2.929134936 podStartE2EDuration="2.929134936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:40.926407762 +0000 UTC m=+1243.858275274" watchObservedRunningTime="2026-11-03 20:17:40.929134936 +0000 UTC m=+1243.861002498" Nov 03 20:17:41.165973 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:41.165938 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:41.171885 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:41.171858 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:41.785098 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:41.785071 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7" Nov 03 20:17:41.901323 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:41.901289 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:41.902251 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:41.902233 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:17:42.847954 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.847894 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:17:42.857240 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.857215 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" Nov 03 20:17:42.912423 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.912383 4918 generic.go:356] "Generic (PLEG): container finished" podID="c878724e-bf1c-4ff5-836b-af4e0304c6dc" containerID="6189419cd87e5bbe6ab9d82da44147773b6f5e09cc95422f860d9ab3d753839d" exitCode=0 Nov 03 20:17:42.912976 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.912442 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" Nov 03 20:17:42.912976 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.912490 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" event={"ID":"c878724e-bf1c-4ff5-836b-af4e0304c6dc","Type":"ContainerDied","Data":"6189419cd87e5bbe6ab9d82da44147773b6f5e09cc95422f860d9ab3d753839d"} Nov 03 20:17:42.912976 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.912569 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf" event={"ID":"c878724e-bf1c-4ff5-836b-af4e0304c6dc","Type":"ContainerDied","Data":"bfa613742d67ea6f0e486c3a98b70463e62aa2d83a88516b981c5000d8a57b12"} Nov 03 20:17:42.912976 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.912685 4918 scope.go:122] "RemoveContainer" containerID="6189419cd87e5bbe6ab9d82da44147773b6f5e09cc95422f860d9ab3d753839d" Nov 03 20:17:42.916705 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.916677 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-credential-socket\") pod \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " Nov 03 20:17:42.916841 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.916722 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istiod-ca-cert\") pod \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " Nov 03 20:17:42.916841 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.916788 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c878724e-bf1c-4ff5-836b-af4e0304c6dc-kube-api-access-rdwd9\" (UniqueName: \"kubernetes.io/projected/c878724e-bf1c-4ff5-836b-af4e0304c6dc-kube-api-access-rdwd9\") pod \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " Nov 03 20:17:42.916841 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.916824 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-podinfo\") pod \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " Nov 03 20:17:42.917016 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.916950 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-data\") pod \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " Nov 03 20:17:42.917016 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.916953 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-credential-socket" pod "c878724e-bf1c-4ff5-836b-af4e0304c6dc" (UID: "c878724e-bf1c-4ff5-836b-af4e0304c6dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:42.917130 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.917050 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-workload-certs\") pod \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " Nov 03 20:17:42.917204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.917139 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-envoy\") pod \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " Nov 03 20:17:42.917260 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.917241 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-workload-socket\") pod \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " Nov 03 20:17:42.917305 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.917270 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-token\" (UniqueName: \"kubernetes.io/projected/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-token\") pod \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\" (UID: \"c878724e-bf1c-4ff5-836b-af4e0304c6dc\") " Nov 03 20:17:42.917546 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.917520 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istiod-ca-cert" pod "c878724e-bf1c-4ff5-836b-af4e0304c6dc" (UID: "c878724e-bf1c-4ff5-836b-af4e0304c6dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:42.918631 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.917865 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-data" pod "c878724e-bf1c-4ff5-836b-af4e0304c6dc" (UID: "c878724e-bf1c-4ff5-836b-af4e0304c6dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:42.918631 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.918233 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:42.918631 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.918257 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:42.918631 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.918271 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:42.919831 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.919521 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c878724e-bf1c-4ff5-836b-af4e0304c6dc-kube-api-access-rdwd9" pod "c878724e-bf1c-4ff5-836b-af4e0304c6dc" (UID: "c878724e-bf1c-4ff5-836b-af4e0304c6dc"). InnerVolumeSpecName "kube-api-access-rdwd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:42.919831 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.919806 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-workload-socket" pod "c878724e-bf1c-4ff5-836b-af4e0304c6dc" (UID: "c878724e-bf1c-4ff5-836b-af4e0304c6dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:42.920665 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.920639 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-workload-certs" pod "c878724e-bf1c-4ff5-836b-af4e0304c6dc" (UID: "c878724e-bf1c-4ff5-836b-af4e0304c6dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:42.927323 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.927195 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-token" pod "c878724e-bf1c-4ff5-836b-af4e0304c6dc" (UID: "c878724e-bf1c-4ff5-836b-af4e0304c6dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:42.927969 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.927487 4918 scope.go:122] "RemoveContainer" containerID="e9a1d2237222b859f6f800faca16bb42f9b604e85ac6ea529af4dea003228638" Nov 03 20:17:42.927969 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.927496 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-podinfo" pod "c878724e-bf1c-4ff5-836b-af4e0304c6dc" (UID: "c878724e-bf1c-4ff5-836b-af4e0304c6dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:17:42.927969 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.927690 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-envoy" pod "c878724e-bf1c-4ff5-836b-af4e0304c6dc" (UID: "c878724e-bf1c-4ff5-836b-af4e0304c6dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:42.981135 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.981107 4918 scope.go:122] "RemoveContainer" containerID="6189419cd87e5bbe6ab9d82da44147773b6f5e09cc95422f860d9ab3d753839d" Nov 03 20:17:42.982314 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:17:42.982282 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6189419cd87e5bbe6ab9d82da44147773b6f5e09cc95422f860d9ab3d753839d\": container with ID starting with 6189419cd87e5bbe6ab9d82da44147773b6f5e09cc95422f860d9ab3d753839d not found: ID does not exist" containerID="6189419cd87e5bbe6ab9d82da44147773b6f5e09cc95422f860d9ab3d753839d" Nov 03 20:17:42.982449 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.982326 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6189419cd87e5bbe6ab9d82da44147773b6f5e09cc95422f860d9ab3d753839d"} err="failed to get container status \"6189419cd87e5bbe6ab9d82da44147773b6f5e09cc95422f860d9ab3d753839d\": rpc error: code = NotFound desc = could not find container \"6189419cd87e5bbe6ab9d82da44147773b6f5e09cc95422f860d9ab3d753839d\": container with ID starting with 6189419cd87e5bbe6ab9d82da44147773b6f5e09cc95422f860d9ab3d753839d not found: ID does not exist" Nov 03 20:17:42.982449 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.982350 4918 scope.go:122] "RemoveContainer" containerID="e9a1d2237222b859f6f800faca16bb42f9b604e85ac6ea529af4dea003228638" Nov 03 20:17:42.982870 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:17:42.982848 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a1d2237222b859f6f800faca16bb42f9b604e85ac6ea529af4dea003228638\": container with ID starting with e9a1d2237222b859f6f800faca16bb42f9b604e85ac6ea529af4dea003228638 not found: ID does not exist" containerID="e9a1d2237222b859f6f800faca16bb42f9b604e85ac6ea529af4dea003228638" Nov 03 20:17:42.982963 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:42.982874 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a1d2237222b859f6f800faca16bb42f9b604e85ac6ea529af4dea003228638"} err="failed to get container status \"e9a1d2237222b859f6f800faca16bb42f9b604e85ac6ea529af4dea003228638\": rpc error: code = NotFound desc = could not find container \"e9a1d2237222b859f6f800faca16bb42f9b604e85ac6ea529af4dea003228638\": container with ID starting with e9a1d2237222b859f6f800faca16bb42f9b604e85ac6ea529af4dea003228638 not found: ID does not exist" Nov 03 20:17:43.020155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:43.020124 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:43.020155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:43.020151 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:43.020376 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:43.020166 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:43.020376 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:43.020188 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdwd9\" (UniqueName: \"kubernetes.io/projected/c878724e-bf1c-4ff5-836b-af4e0304c6dc-kube-api-access-rdwd9\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:43.020376 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:43.020197 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c878724e-bf1c-4ff5-836b-af4e0304c6dc-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:43.020376 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:43.020206 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c878724e-bf1c-4ff5-836b-af4e0304c6dc-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:43.241965 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:43.241929 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf"] Nov 03 20:17:43.245748 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:43.245719 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ffd1cf5a-openshift-default-5b4b868cd5-p2njf"] Nov 03 20:17:44.517108 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.517074 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw"] Nov 03 20:17:44.517752 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.517738 4918 cpu_manager.go:414] "containerMap: removing container" podUID="c878724e-bf1c-4ff5-836b-af4e0304c6dc" containerName="istio-proxy" Nov 03 20:17:44.517796 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.517753 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c878724e-bf1c-4ff5-836b-af4e0304c6dc" containerName="istio-proxy" Nov 03 20:17:44.517892 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.517882 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c878724e-bf1c-4ff5-836b-af4e0304c6dc" containerName="istio-proxy" Nov 03 20:17:44.523000 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.522980 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw" Nov 03 20:17:44.531382 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.531050 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-b35c30\"" Nov 03 20:17:44.531382 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.531291 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-b35c30-dockercfg-rs2m2\"" Nov 03 20:17:44.536534 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.533819 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw"] Nov 03 20:17:44.548590 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.548563 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp"] Nov 03 20:17:44.549318 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.549299 4918 cpu_manager.go:414] "containerMap: removing container" podUID="c878724e-bf1c-4ff5-836b-af4e0304c6dc" containerName="istio-proxy" Nov 03 20:17:44.549423 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.549318 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c878724e-bf1c-4ff5-836b-af4e0304c6dc" containerName="istio-proxy" Nov 03 20:17:44.549497 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.549483 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c878724e-bf1c-4ff5-836b-af4e0304c6dc" containerName="istio-proxy" Nov 03 20:17:44.553381 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.553356 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp" Nov 03 20:17:44.554221 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.554196 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ea5be43-cacb-48ac-89bb-ca19bdc86522-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw\" (UID: \"2ea5be43-cacb-48ac-89bb-ca19bdc86522\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw" Nov 03 20:17:44.554458 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.554435 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26xck\" (UniqueName: \"kubernetes.io/projected/2ea5be43-cacb-48ac-89bb-ca19bdc86522-kube-api-access-26xck\") pod \"payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw\" (UID: \"2ea5be43-cacb-48ac-89bb-ca19bdc86522\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw" Nov 03 20:17:44.564326 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.564301 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp"] Nov 03 20:17:44.716489 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.715358 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d270ff2c-258f-4c66-b50e-1bbb2494816d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp\" (UID: \"d270ff2c-258f-4c66-b50e-1bbb2494816d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp" Nov 03 20:17:44.716489 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.715520 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26xck\" (UniqueName: \"kubernetes.io/projected/2ea5be43-cacb-48ac-89bb-ca19bdc86522-kube-api-access-26xck\") pod \"payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw\" (UID: \"2ea5be43-cacb-48ac-89bb-ca19bdc86522\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw" Nov 03 20:17:44.716489 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.715953 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ea5be43-cacb-48ac-89bb-ca19bdc86522-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw\" (UID: \"2ea5be43-cacb-48ac-89bb-ca19bdc86522\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw" Nov 03 20:17:44.716489 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.716009 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwf9m\" (UniqueName: \"kubernetes.io/projected/d270ff2c-258f-4c66-b50e-1bbb2494816d-kube-api-access-gwf9m\") pod \"payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp\" (UID: \"d270ff2c-258f-4c66-b50e-1bbb2494816d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp" Nov 03 20:17:44.718130 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.717092 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ea5be43-cacb-48ac-89bb-ca19bdc86522-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw\" (UID: \"2ea5be43-cacb-48ac-89bb-ca19bdc86522\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw" Nov 03 20:17:44.755978 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.755908 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26xck\" (UniqueName: \"kubernetes.io/projected/2ea5be43-cacb-48ac-89bb-ca19bdc86522-kube-api-access-26xck\") pod \"payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw\" (UID: \"2ea5be43-cacb-48ac-89bb-ca19bdc86522\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw" Nov 03 20:17:44.821229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.820765 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwf9m\" (UniqueName: \"kubernetes.io/projected/d270ff2c-258f-4c66-b50e-1bbb2494816d-kube-api-access-gwf9m\") pod \"payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp\" (UID: \"d270ff2c-258f-4c66-b50e-1bbb2494816d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp" Nov 03 20:17:44.821229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.821053 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d270ff2c-258f-4c66-b50e-1bbb2494816d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp\" (UID: \"d270ff2c-258f-4c66-b50e-1bbb2494816d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp" Nov 03 20:17:44.836201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.822167 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d270ff2c-258f-4c66-b50e-1bbb2494816d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp\" (UID: \"d270ff2c-258f-4c66-b50e-1bbb2494816d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp" Nov 03 20:17:44.857321 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.842326 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw" Nov 03 20:17:44.857321 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.842369 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwf9m\" (UniqueName: \"kubernetes.io/projected/d270ff2c-258f-4c66-b50e-1bbb2494816d-kube-api-access-gwf9m\") pod \"payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp\" (UID: \"d270ff2c-258f-4c66-b50e-1bbb2494816d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp" Nov 03 20:17:44.873273 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:44.872686 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp" Nov 03 20:17:45.158097 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:45.158067 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw"] Nov 03 20:17:45.159062 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:17:45.159023 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea5be43_cacb_48ac_89bb_ca19bdc86522.slice/crio-6e3cf8c6a6db0fd225f7bdf807a7db7d7ff38bb2c10a5d8dc3a1bedc4dbfa582 WatchSource:0}: Error finding container 6e3cf8c6a6db0fd225f7bdf807a7db7d7ff38bb2c10a5d8dc3a1bedc4dbfa582: Status 404 returned error can't find the container with id 6e3cf8c6a6db0fd225f7bdf807a7db7d7ff38bb2c10a5d8dc3a1bedc4dbfa582 Nov 03 20:17:45.182990 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:17:45.182958 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd270ff2c_258f_4c66_b50e_1bbb2494816d.slice/crio-2f1f513a3cbd18376ad1187566498fc4f7e8dd8dd986590a99def89cc5057972 WatchSource:0}: Error finding container 2f1f513a3cbd18376ad1187566498fc4f7e8dd8dd986590a99def89cc5057972: Status 404 returned error can't find the container with id 2f1f513a3cbd18376ad1187566498fc4f7e8dd8dd986590a99def89cc5057972 Nov 03 20:17:45.183103 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:45.183011 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp"] Nov 03 20:17:45.217775 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:45.217748 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c878724e-bf1c-4ff5-836b-af4e0304c6dc" path="/var/lib/kubelet/pods/c878724e-bf1c-4ff5-836b-af4e0304c6dc/volumes" Nov 03 20:17:45.938754 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:45.938698 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp" event={"ID":"d270ff2c-258f-4c66-b50e-1bbb2494816d","Type":"ContainerStarted","Data":"4b9bcba9e92025c1b16871c76abd5b474d0ecc233a8f84964a273b76085b9002"} Nov 03 20:17:45.939283 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:45.938789 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp" Nov 03 20:17:45.939283 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:45.938832 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp" event={"ID":"d270ff2c-258f-4c66-b50e-1bbb2494816d","Type":"ContainerStarted","Data":"2f1f513a3cbd18376ad1187566498fc4f7e8dd8dd986590a99def89cc5057972"} Nov 03 20:17:45.940704 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:45.940658 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw" event={"ID":"2ea5be43-cacb-48ac-89bb-ca19bdc86522","Type":"ContainerStarted","Data":"1d76475cf7316fc3ccd41796daa6204fee0fe1a6d3e13eb966302f90ef71fb68"} Nov 03 20:17:45.940889 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:45.940705 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw" event={"ID":"2ea5be43-cacb-48ac-89bb-ca19bdc86522","Type":"ContainerStarted","Data":"6e3cf8c6a6db0fd225f7bdf807a7db7d7ff38bb2c10a5d8dc3a1bedc4dbfa582"} Nov 03 20:17:45.940889 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:45.940750 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw" Nov 03 20:17:45.960334 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:45.960290 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp" podStartSLOduration=1.960278745 podStartE2EDuration="1.960278745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:45.957207956 +0000 UTC m=+1248.889075468" watchObservedRunningTime="2026-11-03 20:17:45.960278745 +0000 UTC m=+1248.892146256" Nov 03 20:17:45.980073 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:45.980009 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw" podStartSLOduration=1.9799768850000001 podStartE2EDuration="1.979976885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:45.977691958 +0000 UTC m=+1248.909559471" watchObservedRunningTime="2026-11-03 20:17:45.979976885 +0000 UTC m=+1248.911844397" Nov 03 20:17:56.962466 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:56.962431 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw" Nov 03 20:17:56.963002 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:56.962600 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp" Nov 03 20:17:57.873783 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:57.873752 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj"] Nov 03 20:17:57.879540 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:57.879519 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj" Nov 03 20:17:57.889900 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:57.889869 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-1c6a98-dockercfg-z8mjm\"" Nov 03 20:17:57.893589 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:57.893567 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-1c6a98\"" Nov 03 20:17:57.896923 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:57.896902 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj"] Nov 03 20:17:57.899775 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:57.899747 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d724b00-97ad-4d6f-85a6-3a273b5042f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj\" (UID: \"0d724b00-97ad-4d6f-85a6-3a273b5042f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj" Nov 03 20:17:57.900061 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:57.900033 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzzwl\" (UniqueName: \"kubernetes.io/projected/0d724b00-97ad-4d6f-85a6-3a273b5042f1-kube-api-access-lzzwl\") pod \"payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj\" (UID: \"0d724b00-97ad-4d6f-85a6-3a273b5042f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj" Nov 03 20:17:58.030067 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:58.030023 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d724b00-97ad-4d6f-85a6-3a273b5042f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj\" (UID: \"0d724b00-97ad-4d6f-85a6-3a273b5042f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj" Nov 03 20:17:58.030592 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:58.030241 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzzwl\" (UniqueName: \"kubernetes.io/projected/0d724b00-97ad-4d6f-85a6-3a273b5042f1-kube-api-access-lzzwl\") pod \"payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj\" (UID: \"0d724b00-97ad-4d6f-85a6-3a273b5042f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj" Nov 03 20:17:58.044271 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:58.032555 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d724b00-97ad-4d6f-85a6-3a273b5042f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj\" (UID: \"0d724b00-97ad-4d6f-85a6-3a273b5042f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj" Nov 03 20:17:58.195277 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:58.192939 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzzwl\" (UniqueName: \"kubernetes.io/projected/0d724b00-97ad-4d6f-85a6-3a273b5042f1-kube-api-access-lzzwl\") pod \"payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj\" (UID: \"0d724b00-97ad-4d6f-85a6-3a273b5042f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj" Nov 03 20:17:58.202201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:58.198538 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj" Nov 03 20:17:58.271767 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:17:58.271730 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:17:58.562209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:58.562156 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj"] Nov 03 20:17:58.562405 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:17:58.562376 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d724b00_97ad_4d6f_85a6_3a273b5042f1.slice/crio-58eacd342baf31ec6fc441bffaf63efd6a58afd5ecf10c9b39f97f1bcd56fc65 WatchSource:0}: Error finding container 58eacd342baf31ec6fc441bffaf63efd6a58afd5ecf10c9b39f97f1bcd56fc65: Status 404 returned error can't find the container with id 58eacd342baf31ec6fc441bffaf63efd6a58afd5ecf10c9b39f97f1bcd56fc65 Nov 03 20:17:59.148006 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:59.147965 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj" event={"ID":"0d724b00-97ad-4d6f-85a6-3a273b5042f1","Type":"ContainerStarted","Data":"fb1e582595e217fa0f29700910b824944d09af60ee8ae9d2e711762348d20d6c"} Nov 03 20:17:59.148438 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:59.148029 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj" event={"ID":"0d724b00-97ad-4d6f-85a6-3a273b5042f1","Type":"ContainerStarted","Data":"58eacd342baf31ec6fc441bffaf63efd6a58afd5ecf10c9b39f97f1bcd56fc65"} Nov 03 20:17:59.148438 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:59.148058 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj" Nov 03 20:17:59.167561 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:17:59.167512 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj" podStartSLOduration=2.167498472 podStartE2EDuration="2.167498472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:59.165160075 +0000 UTC m=+1262.097027586" watchObservedRunningTime="2026-11-03 20:17:59.167498472 +0000 UTC m=+1262.099365982" Nov 03 20:18:10.193213 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.193144 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj" Nov 03 20:18:10.673312 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.673271 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl"] Nov 03 20:18:10.683442 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.682753 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl" Nov 03 20:18:10.688091 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.687985 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl"] Nov 03 20:18:10.692043 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.690433 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-ee4c78\"" Nov 03 20:18:10.707405 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.707377 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs"] Nov 03 20:18:10.709866 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.709842 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v28zv\" (UniqueName: \"kubernetes.io/projected/7b781aac-9f06-47eb-ac6a-16ba694919b2-kube-api-access-v28zv\") pod \"payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl\" (UID: \"7b781aac-9f06-47eb-ac6a-16ba694919b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl" Nov 03 20:18:10.710048 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.710035 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b781aac-9f06-47eb-ac6a-16ba694919b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl\" (UID: \"7b781aac-9f06-47eb-ac6a-16ba694919b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl" Nov 03 20:18:10.713051 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.713031 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs" Nov 03 20:18:10.770425 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.770335 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs"] Nov 03 20:18:10.778757 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.778724 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-ee4c78-dockercfg-pkq82\"" Nov 03 20:18:10.811709 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.811675 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b781aac-9f06-47eb-ac6a-16ba694919b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl\" (UID: \"7b781aac-9f06-47eb-ac6a-16ba694919b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl" Nov 03 20:18:10.811863 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.811768 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef65466e-4f77-454d-9089-a7b0c636b563-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs\" (UID: \"ef65466e-4f77-454d-9089-a7b0c636b563\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs" Nov 03 20:18:10.811911 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.811865 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqfr9\" (UniqueName: \"kubernetes.io/projected/ef65466e-4f77-454d-9089-a7b0c636b563-kube-api-access-qqfr9\") pod \"payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs\" (UID: \"ef65466e-4f77-454d-9089-a7b0c636b563\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs" Nov 03 20:18:10.811947 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.811911 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v28zv\" (UniqueName: \"kubernetes.io/projected/7b781aac-9f06-47eb-ac6a-16ba694919b2-kube-api-access-v28zv\") pod \"payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl\" (UID: \"7b781aac-9f06-47eb-ac6a-16ba694919b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl" Nov 03 20:18:10.812851 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.812822 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b781aac-9f06-47eb-ac6a-16ba694919b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl\" (UID: \"7b781aac-9f06-47eb-ac6a-16ba694919b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl" Nov 03 20:18:10.842016 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.841946 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v28zv\" (UniqueName: \"kubernetes.io/projected/7b781aac-9f06-47eb-ac6a-16ba694919b2-kube-api-access-v28zv\") pod \"payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl\" (UID: \"7b781aac-9f06-47eb-ac6a-16ba694919b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl" Nov 03 20:18:10.930254 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.913290 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqfr9\" (UniqueName: \"kubernetes.io/projected/ef65466e-4f77-454d-9089-a7b0c636b563-kube-api-access-qqfr9\") pod \"payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs\" (UID: \"ef65466e-4f77-454d-9089-a7b0c636b563\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs" Nov 03 20:18:10.930254 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.913579 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef65466e-4f77-454d-9089-a7b0c636b563-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs\" (UID: \"ef65466e-4f77-454d-9089-a7b0c636b563\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs" Nov 03 20:18:10.930254 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.914658 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef65466e-4f77-454d-9089-a7b0c636b563-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs\" (UID: \"ef65466e-4f77-454d-9089-a7b0c636b563\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs" Nov 03 20:18:10.993552 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:10.992292 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqfr9\" (UniqueName: \"kubernetes.io/projected/ef65466e-4f77-454d-9089-a7b0c636b563-kube-api-access-qqfr9\") pod \"payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs\" (UID: \"ef65466e-4f77-454d-9089-a7b0c636b563\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs" Nov 03 20:18:11.014732 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:11.014463 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl" Nov 03 20:18:11.100201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:11.082270 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs" Nov 03 20:18:11.603970 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:18:11.603467 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b781aac_9f06_47eb_ac6a_16ba694919b2.slice/crio-2ab727394bbe30a8ee865c54f50127ff6c8bcec36587d259901057bab3e882cc WatchSource:0}: Error finding container 2ab727394bbe30a8ee865c54f50127ff6c8bcec36587d259901057bab3e882cc: Status 404 returned error can't find the container with id 2ab727394bbe30a8ee865c54f50127ff6c8bcec36587d259901057bab3e882cc Nov 03 20:18:11.604306 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:11.604076 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl"] Nov 03 20:18:11.607630 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:18:11.607602 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef65466e_4f77_454d_9089_a7b0c636b563.slice/crio-05a97d6bef5f13a00d40686a5fd1e17c4393b5257b14ec696444f2c74d417741 WatchSource:0}: Error finding container 05a97d6bef5f13a00d40686a5fd1e17c4393b5257b14ec696444f2c74d417741: Status 404 returned error can't find the container with id 05a97d6bef5f13a00d40686a5fd1e17c4393b5257b14ec696444f2c74d417741 Nov 03 20:18:11.616213 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:11.616165 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs"] Nov 03 20:18:12.435732 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:12.435516 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs" event={"ID":"ef65466e-4f77-454d-9089-a7b0c636b563","Type":"ContainerStarted","Data":"da6a647af7b41ab083ffd5d3090a8b64f991caa2fefaa9cbf47b7fb4ab6e3d4c"} Nov 03 20:18:12.435732 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:12.435588 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs" event={"ID":"ef65466e-4f77-454d-9089-a7b0c636b563","Type":"ContainerStarted","Data":"05a97d6bef5f13a00d40686a5fd1e17c4393b5257b14ec696444f2c74d417741"} Nov 03 20:18:12.458205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:12.446558 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl" event={"ID":"7b781aac-9f06-47eb-ac6a-16ba694919b2","Type":"ContainerStarted","Data":"48a3ca9eac738b94dca0445bcd49a9b45d2dd43b7a0b2910c1c12ec3da3d7ebb"} Nov 03 20:18:12.458205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:12.446630 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl" event={"ID":"7b781aac-9f06-47eb-ac6a-16ba694919b2","Type":"ContainerStarted","Data":"2ab727394bbe30a8ee865c54f50127ff6c8bcec36587d259901057bab3e882cc"} Nov 03 20:18:12.458205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:12.447853 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl" Nov 03 20:18:12.531203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:12.524052 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl" podStartSLOduration=2.524027411 podStartE2EDuration="2.524027411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:12.481802195 +0000 UTC m=+1275.413669707" watchObservedRunningTime="2026-11-03 20:18:12.524027411 +0000 UTC m=+1275.455894982" Nov 03 20:18:12.578204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:12.577848 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs" podStartSLOduration=2.577829538 podStartE2EDuration="2.577829538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:12.508572869 +0000 UTC m=+1275.440440384" watchObservedRunningTime="2026-11-03 20:18:12.577829538 +0000 UTC m=+1275.509697051" Nov 03 20:18:13.470202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:13.469159 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs" Nov 03 20:18:17.725577 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:17.725536 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg"] Nov 03 20:18:17.874373 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:17.873463 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg"] Nov 03 20:18:17.874373 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:17.873603 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" Nov 03 20:18:17.902856 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:17.885263 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-ee4c78-serving-cert\"" Nov 03 20:18:17.920747 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:17.920697 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg\" (UID: \"c7a163ac-b7b9-460f-a0e5-73faac7e3f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" Nov 03 20:18:17.921039 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:17.921012 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg\" (UID: \"c7a163ac-b7b9-460f-a0e5-73faac7e3f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" Nov 03 20:18:17.921162 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:17.921148 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg\" (UID: \"c7a163ac-b7b9-460f-a0e5-73faac7e3f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" Nov 03 20:18:17.921285 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:17.921203 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mql7\" (UniqueName: \"kubernetes.io/projected/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-kube-api-access-9mql7\") pod \"maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg\" (UID: \"c7a163ac-b7b9-460f-a0e5-73faac7e3f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" Nov 03 20:18:18.023297 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:18.023205 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg\" (UID: \"c7a163ac-b7b9-460f-a0e5-73faac7e3f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" Nov 03 20:18:18.023453 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:18.023362 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg\" (UID: \"c7a163ac-b7b9-460f-a0e5-73faac7e3f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" Nov 03 20:18:18.023453 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:18.023401 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mql7\" (UniqueName: \"kubernetes.io/projected/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-kube-api-access-9mql7\") pod \"maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg\" (UID: \"c7a163ac-b7b9-460f-a0e5-73faac7e3f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" Nov 03 20:18:18.023537 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:18.023516 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg\" (UID: \"c7a163ac-b7b9-460f-a0e5-73faac7e3f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" Nov 03 20:18:18.024248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:18.024163 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg\" (UID: \"c7a163ac-b7b9-460f-a0e5-73faac7e3f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" Nov 03 20:18:18.026028 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:18.026002 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg\" (UID: \"c7a163ac-b7b9-460f-a0e5-73faac7e3f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" Nov 03 20:18:18.026273 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:18.026250 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg\" (UID: \"c7a163ac-b7b9-460f-a0e5-73faac7e3f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" Nov 03 20:18:18.035344 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:18.035321 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mql7\" (UniqueName: \"kubernetes.io/projected/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-kube-api-access-9mql7\") pod \"maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg\" (UID: \"c7a163ac-b7b9-460f-a0e5-73faac7e3f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" Nov 03 20:18:18.169711 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:18.169676 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" Nov 03 20:18:18.640549 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:18.636198 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg"] Nov 03 20:18:18.640912 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:18:18.640813 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a163ac_b7b9_460f_a0e5_73faac7e3f48.slice/crio-ecc15594203910b0787fce888b721fbc750d3fe4ba5b00206ac5b4cc65895c29 WatchSource:0}: Error finding container ecc15594203910b0787fce888b721fbc750d3fe4ba5b00206ac5b4cc65895c29: Status 404 returned error can't find the container with id ecc15594203910b0787fce888b721fbc750d3fe4ba5b00206ac5b4cc65895c29 Nov 03 20:18:19.593256 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:19.593172 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" event={"ID":"c7a163ac-b7b9-460f-a0e5-73faac7e3f48","Type":"ContainerStarted","Data":"a5496cbb9192787365551b802157912e61ebd891e143d70fe059f69deb757b39"} Nov 03 20:18:19.593645 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:19.593292 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" event={"ID":"c7a163ac-b7b9-460f-a0e5-73faac7e3f48","Type":"ContainerStarted","Data":"ecc15594203910b0787fce888b721fbc750d3fe4ba5b00206ac5b4cc65895c29"} Nov 03 20:18:19.593645 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:19.593331 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" Nov 03 20:18:19.634202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:19.618223 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" podStartSLOduration=2.06486933 podStartE2EDuration="2.618202076s" totalImagesPullingTime="553.332746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:19.614710194 +0000 UTC m=+1282.546577708" watchObservedRunningTime="2026-11-03 20:18:19.618202076 +0000 UTC m=+1282.550069588" Nov 03 20:18:21.608642 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:21.608607 4918 generic.go:356] "Generic (PLEG): container finished" podID="9fb1ff44-4b90-47da-a252-d5771f160330" containerID="82bc338730c7f8bd7bd15db32eb5bf81f54b601a7e1da65350bcef80e6b9a38f" exitCode=0 Nov 03 20:18:21.609104 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:21.608686 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" event={"ID":"9fb1ff44-4b90-47da-a252-d5771f160330","Type":"ContainerDied","Data":"82bc338730c7f8bd7bd15db32eb5bf81f54b601a7e1da65350bcef80e6b9a38f"} Nov 03 20:18:21.609320 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:21.609301 4918 scope.go:122] "RemoveContainer" containerID="82bc338730c7f8bd7bd15db32eb5bf81f54b601a7e1da65350bcef80e6b9a38f" Nov 03 20:18:21.610596 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:21.610577 4918 generic.go:356] "Generic (PLEG): container finished" podID="af357454-5d01-46c9-a1ef-bf09169443ee" containerID="20d4b76c2686d47d81d337cd1b4d9edd311a9e9e0c4da64bca25df990eea9a50" exitCode=0 Nov 03 20:18:21.610677 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:21.610655 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" event={"ID":"af357454-5d01-46c9-a1ef-bf09169443ee","Type":"ContainerDied","Data":"20d4b76c2686d47d81d337cd1b4d9edd311a9e9e0c4da64bca25df990eea9a50"} Nov 03 20:18:21.611032 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:21.611018 4918 scope.go:122] "RemoveContainer" containerID="20d4b76c2686d47d81d337cd1b4d9edd311a9e9e0c4da64bca25df990eea9a50" Nov 03 20:18:21.611080 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:21.611023 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:18:21.611114 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:21.611093 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:18:21.611150 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:21.611124 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:18:21.612544 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:21.612511 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:18:21.612655 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:21.612576 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:18:21.612655 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:21.612611 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:18:22.648848 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:22.648807 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" event={"ID":"9fb1ff44-4b90-47da-a252-d5771f160330","Type":"ContainerStarted","Data":"e40f2bef5ce3d5d70256258930c28e79d505eb14701844c3b3e3f3e854e7110e"} Nov 03 20:18:22.667640 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:22.667604 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" event={"ID":"af357454-5d01-46c9-a1ef-bf09169443ee","Type":"ContainerStarted","Data":"adb53663e85dc17295b9550847c3c7721a0e6c4c8ee16cd2bc7da740973482c9"} Nov 03 20:18:23.170757 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:23.170722 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:18:23.182690 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:23.182308 4918 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:18:23.182690 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:23.182401 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" podUID="9fb1ff44-4b90-47da-a252-d5771f160330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:18:23.580144 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:23.580109 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:18:23.581446 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:23.581415 4918 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:18:23.581573 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:23.581467 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" podUID="af357454-5d01-46c9-a1ef-bf09169443ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 20:18:24.167790 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:24.167749 4918 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:18:24.168489 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:24.168447 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" podUID="9fb1ff44-4b90-47da-a252-d5771f160330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:18:24.478606 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:24.478524 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs" Nov 03 20:18:24.478606 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:24.478595 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl" Nov 03 20:18:24.581003 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:24.580950 4918 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:18:24.581335 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:24.581290 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" podUID="af357454-5d01-46c9-a1ef-bf09169443ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 20:18:25.166261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:25.166222 4918 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:18:25.166554 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:25.166523 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" podUID="9fb1ff44-4b90-47da-a252-d5771f160330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:18:25.581937 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:25.580642 4918 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:18:25.581937 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:25.580687 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" podUID="af357454-5d01-46c9-a1ef-bf09169443ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 20:18:25.596728 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:25.596700 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:18:25.606082 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:25.606054 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" Nov 03 20:18:26.172631 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:26.172586 4918 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:18:26.172971 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:26.172941 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" podUID="9fb1ff44-4b90-47da-a252-d5771f160330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:18:26.580475 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:26.580439 4918 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:18:26.580778 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:26.580735 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" podUID="af357454-5d01-46c9-a1ef-bf09169443ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 20:18:26.901921 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:26.901840 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:18:27.165667 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:27.165571 4918 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:18:27.165931 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:27.165902 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" podUID="9fb1ff44-4b90-47da-a252-d5771f160330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:18:27.581670 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:27.581619 4918 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:18:27.581895 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:27.581685 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" podUID="af357454-5d01-46c9-a1ef-bf09169443ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 20:18:28.165792 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:28.165757 4918 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:18:28.166220 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:28.165807 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" podUID="9fb1ff44-4b90-47da-a252-d5771f160330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:18:28.212759 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:18:28.212665 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:18:28.585589 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:28.585557 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:18:28.587602 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:28.587575 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:18:29.169997 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:29.169968 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:18:29.171277 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:29.171253 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:18:30.338751 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:30.338708 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7084-c20e94"] Nov 03 20:18:30.343322 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:30.343297 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-c20e94" Nov 03 20:18:30.347407 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:30.347387 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dwjcb\"" Nov 03 20:18:30.350074 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:30.350053 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7084-c20e94"] Nov 03 20:18:30.479929 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:30.479886 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l9zm\" (UniqueName: \"kubernetes.io/projected/1384f505-f569-4e54-9df2-eafb20302452-kube-api-access-4l9zm\") pod \"test-curl-7084-c20e94\" (UID: \"1384f505-f569-4e54-9df2-eafb20302452\") " pod="opendatahub/test-curl-7084-c20e94" Nov 03 20:18:30.581836 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:30.581789 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l9zm\" (UniqueName: \"kubernetes.io/projected/1384f505-f569-4e54-9df2-eafb20302452-kube-api-access-4l9zm\") pod \"test-curl-7084-c20e94\" (UID: \"1384f505-f569-4e54-9df2-eafb20302452\") " pod="opendatahub/test-curl-7084-c20e94" Nov 03 20:18:30.593812 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:30.593742 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l9zm\" (UniqueName: \"kubernetes.io/projected/1384f505-f569-4e54-9df2-eafb20302452-kube-api-access-4l9zm\") pod \"test-curl-7084-c20e94\" (UID: \"1384f505-f569-4e54-9df2-eafb20302452\") " pod="opendatahub/test-curl-7084-c20e94" Nov 03 20:18:30.672347 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:30.672313 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-c20e94" Nov 03 20:18:31.034773 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:18:31.034744 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1384f505_f569_4e54_9df2_eafb20302452.slice/crio-e1b46a4f6e02e3156a8056f2cec8931ad1ecb0472e1d616c0580543f431daa26 WatchSource:0}: Error finding container e1b46a4f6e02e3156a8056f2cec8931ad1ecb0472e1d616c0580543f431daa26: Status 404 returned error can't find the container with id e1b46a4f6e02e3156a8056f2cec8931ad1ecb0472e1d616c0580543f431daa26 Nov 03 20:18:31.034921 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:31.034886 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7084-c20e94"] Nov 03 20:18:31.936914 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:31.936876 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7084-c20e94" event={"ID":"1384f505-f569-4e54-9df2-eafb20302452","Type":"ContainerStarted","Data":"e1b46a4f6e02e3156a8056f2cec8931ad1ecb0472e1d616c0580543f431daa26"} Nov 03 20:18:34.982749 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:34.982711 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7084-c20e94" event={"ID":"1384f505-f569-4e54-9df2-eafb20302452","Type":"ContainerStarted","Data":"7c2a2cbba07b5deea310660ff0b5c08e828a5d135e53ce82911b68f6a32f85f5"} Nov 03 20:18:35.000392 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:35.000341 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7084-c20e94" podStartSLOduration=1.939291594 podStartE2EDuration="5.000325499s" totalImagesPullingTime="3.061033905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:34.997707424 +0000 UTC m=+1297.929574936" watchObservedRunningTime="2026-11-03 20:18:35.000325499 +0000 UTC m=+1297.932193010" Nov 03 20:18:35.696310 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:35.696264 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7084-c20e94"] Nov 03 20:18:35.702074 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:35.702041 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7084-c20e94"] Nov 03 20:18:35.848026 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:35.847989 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7084-9c86f5"] Nov 03 20:18:35.848775 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:35.848758 4918 cpu_manager.go:414] "containerMap: removing container" podUID="1384f505-f569-4e54-9df2-eafb20302452" containerName="test-curl-7084-c20e94" Nov 03 20:18:35.848828 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:35.848777 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1384f505-f569-4e54-9df2-eafb20302452" containerName="test-curl-7084-c20e94" Nov 03 20:18:35.848899 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:35.848889 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1384f505-f569-4e54-9df2-eafb20302452" containerName="test-curl-7084-c20e94" Nov 03 20:18:35.852777 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:35.852761 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-9c86f5" Nov 03 20:18:35.860021 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:35.859986 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7084-9c86f5"] Nov 03 20:18:35.934303 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:35.934261 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9pdv\" (UniqueName: \"kubernetes.io/projected/815b568b-8ea9-4617-a24c-7aed31412313-kube-api-access-n9pdv\") pod \"test-curl-7084-9c86f5\" (UID: \"815b568b-8ea9-4617-a24c-7aed31412313\") " pod="opendatahub/test-curl-7084-9c86f5" Nov 03 20:18:36.036084 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:36.035987 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9pdv\" (UniqueName: \"kubernetes.io/projected/815b568b-8ea9-4617-a24c-7aed31412313-kube-api-access-n9pdv\") pod \"test-curl-7084-9c86f5\" (UID: \"815b568b-8ea9-4617-a24c-7aed31412313\") " pod="opendatahub/test-curl-7084-9c86f5" Nov 03 20:18:36.048455 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:36.048423 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9pdv\" (UniqueName: \"kubernetes.io/projected/815b568b-8ea9-4617-a24c-7aed31412313-kube-api-access-n9pdv\") pod \"test-curl-7084-9c86f5\" (UID: \"815b568b-8ea9-4617-a24c-7aed31412313\") " pod="opendatahub/test-curl-7084-9c86f5" Nov 03 20:18:36.169501 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:36.169460 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-9c86f5" Nov 03 20:18:36.614891 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:36.611749 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7084-9c86f5"] Nov 03 20:18:37.050738 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:37.050633 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7084-c20e94" podUID="1384f505-f569-4e54-9df2-eafb20302452" containerName="test-curl-7084-c20e94" containerID="cri-o://7c2a2cbba07b5deea310660ff0b5c08e828a5d135e53ce82911b68f6a32f85f5" gracePeriod=2 Nov 03 20:18:37.051251 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:37.051006 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7084-9c86f5" event={"ID":"815b568b-8ea9-4617-a24c-7aed31412313","Type":"ContainerStarted","Data":"2645e49243c122ec357a0baa62dbbaa3bdcd3f8e55d32f8bb951c349c6061ca3"} Nov 03 20:18:37.051251 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:37.051052 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7084-9c86f5" event={"ID":"815b568b-8ea9-4617-a24c-7aed31412313","Type":"ContainerStarted","Data":"3d0ae9f6d314af0a5d46de9a3015ccf8d43bc8344fa5f2965d5fb5f0b0491dd4"} Nov 03 20:18:37.098715 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:37.098648 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7084-9c86f5" podStartSLOduration=2.098628561 podStartE2EDuration="2.098628561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:37.097652345 +0000 UTC m=+1300.029519857" watchObservedRunningTime="2026-11-03 20:18:37.098628561 +0000 UTC m=+1300.030496073" Nov 03 20:18:37.100665 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:18:37.100624 4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7084-c20e94\" is forbidden: User \"system:node:ip-10-0-139-53.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.us-east-2.compute.internal' and this object" podUID="1384f505-f569-4e54-9df2-eafb20302452" pod="opendatahub/test-curl-7084-c20e94" Nov 03 20:18:37.103733 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:18:37.103692 4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7084-c20e94\" is forbidden: User \"system:node:ip-10-0-139-53.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.us-east-2.compute.internal' and this object" podUID="1384f505-f569-4e54-9df2-eafb20302452" pod="opendatahub/test-curl-7084-c20e94" Nov 03 20:18:37.227200 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:18:37.221845 4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7084-c20e94\" is forbidden: User \"system:node:ip-10-0-139-53.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.us-east-2.compute.internal' and this object" podUID="1384f505-f569-4e54-9df2-eafb20302452" pod="opendatahub/test-curl-7084-c20e94" Nov 03 20:18:38.097844 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:38.097757 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7084-9c86f5"] Nov 03 20:18:38.102648 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:38.102615 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7084-9c86f5"] Nov 03 20:18:38.254371 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:38.254339 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7084-98918b"] Nov 03 20:18:38.255080 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:38.255063 4918 cpu_manager.go:414] "containerMap: removing container" podUID="815b568b-8ea9-4617-a24c-7aed31412313" containerName="test-curl-7084-9c86f5" Nov 03 20:18:38.255135 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:38.255081 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815b568b-8ea9-4617-a24c-7aed31412313" containerName="test-curl-7084-9c86f5" Nov 03 20:18:38.255261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:38.255251 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="815b568b-8ea9-4617-a24c-7aed31412313" containerName="test-curl-7084-9c86f5" Nov 03 20:18:38.260005 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:38.259979 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-98918b" Nov 03 20:18:38.264512 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:38.264490 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7084-98918b"] Nov 03 20:18:38.402707 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:38.402609 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgpw8\" (UniqueName: \"kubernetes.io/projected/45f3cc24-f74a-4098-a9bf-8a5fb51ce793-kube-api-access-bgpw8\") pod \"test-curl-7084-98918b\" (UID: \"45f3cc24-f74a-4098-a9bf-8a5fb51ce793\") " pod="opendatahub/test-curl-7084-98918b" Nov 03 20:18:38.529078 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:38.529044 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgpw8\" (UniqueName: \"kubernetes.io/projected/45f3cc24-f74a-4098-a9bf-8a5fb51ce793-kube-api-access-bgpw8\") pod \"test-curl-7084-98918b\" (UID: \"45f3cc24-f74a-4098-a9bf-8a5fb51ce793\") " pod="opendatahub/test-curl-7084-98918b" Nov 03 20:18:38.541800 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:38.541770 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgpw8\" (UniqueName: \"kubernetes.io/projected/45f3cc24-f74a-4098-a9bf-8a5fb51ce793-kube-api-access-bgpw8\") pod \"test-curl-7084-98918b\" (UID: \"45f3cc24-f74a-4098-a9bf-8a5fb51ce793\") " pod="opendatahub/test-curl-7084-98918b" Nov 03 20:18:38.576649 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:38.576615 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-98918b" Nov 03 20:18:38.869406 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:38.869369 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7084-98918b"] Nov 03 20:18:38.874460 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:18:38.874432 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45f3cc24_f74a_4098_a9bf_8a5fb51ce793.slice/crio-a50c7eebc8ab78898b84e0a3bfe0dfcd1abd6b7c757692c9d93fa3815e1c2d7f WatchSource:0}: Error finding container a50c7eebc8ab78898b84e0a3bfe0dfcd1abd6b7c757692c9d93fa3815e1c2d7f: Status 404 returned error can't find the container with id a50c7eebc8ab78898b84e0a3bfe0dfcd1abd6b7c757692c9d93fa3815e1c2d7f Nov 03 20:18:39.070564 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:39.070520 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7084-98918b" event={"ID":"45f3cc24-f74a-4098-a9bf-8a5fb51ce793","Type":"ContainerStarted","Data":"202924e2ee37e83f57a205771c38c67ecab79601d26d125e84c39854fea32eec"} Nov 03 20:18:39.070668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:39.070575 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7084-98918b" event={"ID":"45f3cc24-f74a-4098-a9bf-8a5fb51ce793","Type":"ContainerStarted","Data":"a50c7eebc8ab78898b84e0a3bfe0dfcd1abd6b7c757692c9d93fa3815e1c2d7f"} Nov 03 20:18:39.070668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:39.070634 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7084-9c86f5" podUID="815b568b-8ea9-4617-a24c-7aed31412313" containerName="test-curl-7084-9c86f5" containerID="cri-o://2645e49243c122ec357a0baa62dbbaa3bdcd3f8e55d32f8bb951c349c6061ca3" gracePeriod=2 Nov 03 20:18:39.073891 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:18:39.073858 4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7084-9c86f5\" is forbidden: User \"system:node:ip-10-0-139-53.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.us-east-2.compute.internal' and this object" podUID="815b568b-8ea9-4617-a24c-7aed31412313" pod="opendatahub/test-curl-7084-9c86f5" Nov 03 20:18:39.095772 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:18:39.095736 4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7084-9c86f5\" is forbidden: User \"system:node:ip-10-0-139-53.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.us-east-2.compute.internal' and this object" podUID="815b568b-8ea9-4617-a24c-7aed31412313" pod="opendatahub/test-curl-7084-9c86f5" Nov 03 20:18:39.097749 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:39.097692 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7084-98918b" podStartSLOduration=1.09767633 podStartE2EDuration="1.09767633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:39.090940952 +0000 UTC m=+1302.022808463" watchObservedRunningTime="2026-11-03 20:18:39.09767633 +0000 UTC m=+1302.029543844" Nov 03 20:18:39.230119 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:39.230088 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-c20e94" Nov 03 20:18:39.345965 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:39.345863 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1384f505-f569-4e54-9df2-eafb20302452-kube-api-access-4l9zm\" (UniqueName: \"kubernetes.io/projected/1384f505-f569-4e54-9df2-eafb20302452-kube-api-access-4l9zm\") pod \"1384f505-f569-4e54-9df2-eafb20302452\" (UID: \"1384f505-f569-4e54-9df2-eafb20302452\") " Nov 03 20:18:39.348462 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:39.348432 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1384f505-f569-4e54-9df2-eafb20302452-kube-api-access-4l9zm" pod "1384f505-f569-4e54-9df2-eafb20302452" (UID: "1384f505-f569-4e54-9df2-eafb20302452"). InnerVolumeSpecName "kube-api-access-4l9zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:39.453204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:39.452008 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l9zm\" (UniqueName: \"kubernetes.io/projected/1384f505-f569-4e54-9df2-eafb20302452-kube-api-access-4l9zm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:39.843202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:39.842236 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7084-98918b"] Nov 03 20:18:39.892225 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:39.890930 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7084-98918b"] Nov 03 20:18:40.138259 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:40.073288 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7084-8c1590"] Nov 03 20:18:40.138259 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:40.101951 4918 cpu_manager.go:414] "containerMap: removing container" podUID="45f3cc24-f74a-4098-a9bf-8a5fb51ce793" containerName="test-curl-7084-98918b" Nov 03 20:18:40.138259 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:40.101992 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f3cc24-f74a-4098-a9bf-8a5fb51ce793" containerName="test-curl-7084-98918b" Nov 03 20:18:40.138259 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:40.115308 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f3cc24-f74a-4098-a9bf-8a5fb51ce793" containerName="test-curl-7084-98918b" Nov 03 20:18:40.213768 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:40.213740 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7084-8c1590"] Nov 03 20:18:40.215233 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:40.215162 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-8c1590" Nov 03 20:18:40.247204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:40.243432 4918 generic.go:356] "Generic (PLEG): container finished" podID="1384f505-f569-4e54-9df2-eafb20302452" containerID="7c2a2cbba07b5deea310660ff0b5c08e828a5d135e53ce82911b68f6a32f85f5" exitCode=137 Nov 03 20:18:40.247204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:40.244458 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-c20e94" Nov 03 20:18:40.247204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:40.245763 4918 scope.go:122] "RemoveContainer" containerID="7c2a2cbba07b5deea310660ff0b5c08e828a5d135e53ce82911b68f6a32f85f5" Nov 03 20:18:40.309728 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:18:40.309690 4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7084-c20e94\" is forbidden: User \"system:node:ip-10-0-139-53.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.us-east-2.compute.internal' and this object" podUID="1384f505-f569-4e54-9df2-eafb20302452" pod="opendatahub/test-curl-7084-c20e94" Nov 03 20:18:40.311582 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:40.311563 4918 scope.go:122] "RemoveContainer" containerID="7c2a2cbba07b5deea310660ff0b5c08e828a5d135e53ce82911b68f6a32f85f5" Nov 03 20:18:40.311916 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:18:40.311898 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2a2cbba07b5deea310660ff0b5c08e828a5d135e53ce82911b68f6a32f85f5\": container with ID starting with 7c2a2cbba07b5deea310660ff0b5c08e828a5d135e53ce82911b68f6a32f85f5 not found: ID does not exist" containerID="7c2a2cbba07b5deea310660ff0b5c08e828a5d135e53ce82911b68f6a32f85f5" Nov 03 20:18:40.311971 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:40.311925 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2a2cbba07b5deea310660ff0b5c08e828a5d135e53ce82911b68f6a32f85f5"} err="failed to get container status \"7c2a2cbba07b5deea310660ff0b5c08e828a5d135e53ce82911b68f6a32f85f5\": rpc error: code = NotFound desc = could not find container \"7c2a2cbba07b5deea310660ff0b5c08e828a5d135e53ce82911b68f6a32f85f5\": container with ID starting with 7c2a2cbba07b5deea310660ff0b5c08e828a5d135e53ce82911b68f6a32f85f5 not found: ID does not exist" Nov 03 20:18:40.313755 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:40.313724 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcvw7\" (UniqueName: \"kubernetes.io/projected/68d5c814-f1b6-4101-b5a4-6ab08d335af0-kube-api-access-gcvw7\") pod \"test-curl-7084-8c1590\" (UID: \"68d5c814-f1b6-4101-b5a4-6ab08d335af0\") " pod="opendatahub/test-curl-7084-8c1590" Nov 03 20:18:40.416622 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:40.416527 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcvw7\" (UniqueName: \"kubernetes.io/projected/68d5c814-f1b6-4101-b5a4-6ab08d335af0-kube-api-access-gcvw7\") pod \"test-curl-7084-8c1590\" (UID: \"68d5c814-f1b6-4101-b5a4-6ab08d335af0\") " pod="opendatahub/test-curl-7084-8c1590" Nov 03 20:18:40.440136 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:40.440097 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcvw7\" (UniqueName: \"kubernetes.io/projected/68d5c814-f1b6-4101-b5a4-6ab08d335af0-kube-api-access-gcvw7\") pod \"test-curl-7084-8c1590\" (UID: \"68d5c814-f1b6-4101-b5a4-6ab08d335af0\") " pod="opendatahub/test-curl-7084-8c1590" Nov 03 20:18:40.609628 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:40.609598 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-8c1590" Nov 03 20:18:41.310243 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:41.307950 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1384f505-f569-4e54-9df2-eafb20302452" path="/var/lib/kubelet/pods/1384f505-f569-4e54-9df2-eafb20302452/volumes" Nov 03 20:18:41.402597 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:41.402309 4918 generic.go:356] "Generic (PLEG): container finished" podID="815b568b-8ea9-4617-a24c-7aed31412313" containerID="2645e49243c122ec357a0baa62dbbaa3bdcd3f8e55d32f8bb951c349c6061ca3" exitCode=137 Nov 03 20:18:41.404033 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:41.404003 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7084-98918b" podUID="45f3cc24-f74a-4098-a9bf-8a5fb51ce793" containerName="test-curl-7084-98918b" containerID="cri-o://202924e2ee37e83f57a205771c38c67ecab79601d26d125e84c39854fea32eec" gracePeriod=2 Nov 03 20:18:41.575821 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:41.573804 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7084-8c1590"] Nov 03 20:18:41.703520 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:41.703487 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-9c86f5" Nov 03 20:18:41.869355 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:41.869322 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/815b568b-8ea9-4617-a24c-7aed31412313-kube-api-access-n9pdv\" (UniqueName: \"kubernetes.io/projected/815b568b-8ea9-4617-a24c-7aed31412313-kube-api-access-n9pdv\") pod \"815b568b-8ea9-4617-a24c-7aed31412313\" (UID: \"815b568b-8ea9-4617-a24c-7aed31412313\") " Nov 03 20:18:41.871482 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:41.871460 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815b568b-8ea9-4617-a24c-7aed31412313-kube-api-access-n9pdv" pod "815b568b-8ea9-4617-a24c-7aed31412313" (UID: "815b568b-8ea9-4617-a24c-7aed31412313"). InnerVolumeSpecName "kube-api-access-n9pdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:41.992029 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:41.991924 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9pdv\" (UniqueName: \"kubernetes.io/projected/815b568b-8ea9-4617-a24c-7aed31412313-kube-api-access-n9pdv\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:42.412308 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:42.412269 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7084-8c1590" event={"ID":"68d5c814-f1b6-4101-b5a4-6ab08d335af0","Type":"ContainerStarted","Data":"f9fdaefac79529b2d4457c98af2b533e68c233598e40575d2d5f7f7b9ff4f823"} Nov 03 20:18:42.412757 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:42.412317 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7084-8c1590" event={"ID":"68d5c814-f1b6-4101-b5a4-6ab08d335af0","Type":"ContainerStarted","Data":"2188769702f6802d0a57546548fb30e0b5f671bcba1ef2880efabff3707ba824"} Nov 03 20:18:42.413648 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:42.413629 4918 scope.go:122] "RemoveContainer" containerID="2645e49243c122ec357a0baa62dbbaa3bdcd3f8e55d32f8bb951c349c6061ca3" Nov 03 20:18:42.413648 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:42.413639 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-9c86f5" Nov 03 20:18:42.430615 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:18:42.430583 4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7084-9c86f5\" is forbidden: User \"system:node:ip-10-0-139-53.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.us-east-2.compute.internal' and this object" podUID="815b568b-8ea9-4617-a24c-7aed31412313" pod="opendatahub/test-curl-7084-9c86f5" Nov 03 20:18:42.432185 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:42.432140 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7084-8c1590" podStartSLOduration=3.432126549 podStartE2EDuration="3.432126549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:42.42786961 +0000 UTC m=+1305.359737123" watchObservedRunningTime="2026-11-03 20:18:42.432126549 +0000 UTC m=+1305.363994059" Nov 03 20:18:43.192027 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:43.191992 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7084-8c1590"] Nov 03 20:18:43.205252 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:43.205221 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7084-8c1590"] Nov 03 20:18:43.216962 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:43.216931 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="815b568b-8ea9-4617-a24c-7aed31412313" path="/var/lib/kubelet/pods/815b568b-8ea9-4617-a24c-7aed31412313/volumes" Nov 03 20:18:43.645687 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:43.645658 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-98918b" Nov 03 20:18:43.718699 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:43.718606 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f3cc24-f74a-4098-a9bf-8a5fb51ce793-kube-api-access-bgpw8\" (UniqueName: \"kubernetes.io/projected/45f3cc24-f74a-4098-a9bf-8a5fb51ce793-kube-api-access-bgpw8\") pod \"45f3cc24-f74a-4098-a9bf-8a5fb51ce793\" (UID: \"45f3cc24-f74a-4098-a9bf-8a5fb51ce793\") " Nov 03 20:18:43.721087 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:43.721058 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f3cc24-f74a-4098-a9bf-8a5fb51ce793-kube-api-access-bgpw8" pod "45f3cc24-f74a-4098-a9bf-8a5fb51ce793" (UID: "45f3cc24-f74a-4098-a9bf-8a5fb51ce793"). InnerVolumeSpecName "kube-api-access-bgpw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:43.821245 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:43.821207 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgpw8\" (UniqueName: \"kubernetes.io/projected/45f3cc24-f74a-4098-a9bf-8a5fb51ce793-kube-api-access-bgpw8\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:44.438304 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:44.438267 4918 generic.go:356] "Generic (PLEG): container finished" podID="45f3cc24-f74a-4098-a9bf-8a5fb51ce793" containerID="202924e2ee37e83f57a205771c38c67ecab79601d26d125e84c39854fea32eec" exitCode=137 Nov 03 20:18:44.438512 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:44.438341 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-98918b" Nov 03 20:18:44.438512 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:44.438363 4918 scope.go:122] "RemoveContainer" containerID="202924e2ee37e83f57a205771c38c67ecab79601d26d125e84c39854fea32eec" Nov 03 20:18:44.438702 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:44.438659 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7084-8c1590" podUID="68d5c814-f1b6-4101-b5a4-6ab08d335af0" containerName="test-curl-7084-8c1590" containerID="cri-o://f9fdaefac79529b2d4457c98af2b533e68c233598e40575d2d5f7f7b9ff4f823" gracePeriod=2 Nov 03 20:18:44.448778 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:44.448757 4918 scope.go:122] "RemoveContainer" containerID="202924e2ee37e83f57a205771c38c67ecab79601d26d125e84c39854fea32eec" Nov 03 20:18:44.449025 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:18:44.449007 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"202924e2ee37e83f57a205771c38c67ecab79601d26d125e84c39854fea32eec\": container with ID starting with 202924e2ee37e83f57a205771c38c67ecab79601d26d125e84c39854fea32eec not found: ID does not exist" containerID="202924e2ee37e83f57a205771c38c67ecab79601d26d125e84c39854fea32eec" Nov 03 20:18:44.449105 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:44.449034 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"202924e2ee37e83f57a205771c38c67ecab79601d26d125e84c39854fea32eec"} err="failed to get container status \"202924e2ee37e83f57a205771c38c67ecab79601d26d125e84c39854fea32eec\": rpc error: code = NotFound desc = could not find container \"202924e2ee37e83f57a205771c38c67ecab79601d26d125e84c39854fea32eec\": container with ID starting with 202924e2ee37e83f57a205771c38c67ecab79601d26d125e84c39854fea32eec not found: ID does not exist" Nov 03 20:18:44.450537 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:18:44.450512 4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7084-98918b\" is forbidden: User \"system:node:ip-10-0-139-53.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.us-east-2.compute.internal' and this object" podUID="45f3cc24-f74a-4098-a9bf-8a5fb51ce793" pod="opendatahub/test-curl-7084-98918b" Nov 03 20:18:45.217657 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:45.217617 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45f3cc24-f74a-4098-a9bf-8a5fb51ce793" path="/var/lib/kubelet/pods/45f3cc24-f74a-4098-a9bf-8a5fb51ce793/volumes" Nov 03 20:18:46.732526 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:46.732493 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-8c1590" Nov 03 20:18:46.802981 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:46.802949 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d5c814-f1b6-4101-b5a4-6ab08d335af0-kube-api-access-gcvw7\" (UniqueName: \"kubernetes.io/projected/68d5c814-f1b6-4101-b5a4-6ab08d335af0-kube-api-access-gcvw7\") pod \"68d5c814-f1b6-4101-b5a4-6ab08d335af0\" (UID: \"68d5c814-f1b6-4101-b5a4-6ab08d335af0\") " Nov 03 20:18:46.805310 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:46.805285 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d5c814-f1b6-4101-b5a4-6ab08d335af0-kube-api-access-gcvw7" pod "68d5c814-f1b6-4101-b5a4-6ab08d335af0" (UID: "68d5c814-f1b6-4101-b5a4-6ab08d335af0"). InnerVolumeSpecName "kube-api-access-gcvw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:46.905515 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:46.905477 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcvw7\" (UniqueName: \"kubernetes.io/projected/68d5c814-f1b6-4101-b5a4-6ab08d335af0-kube-api-access-gcvw7\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:47.218689 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:47.218656 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d5c814-f1b6-4101-b5a4-6ab08d335af0" path="/var/lib/kubelet/pods/68d5c814-f1b6-4101-b5a4-6ab08d335af0/volumes" Nov 03 20:18:47.464054 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:47.464014 4918 generic.go:356] "Generic (PLEG): container finished" podID="68d5c814-f1b6-4101-b5a4-6ab08d335af0" containerID="f9fdaefac79529b2d4457c98af2b533e68c233598e40575d2d5f7f7b9ff4f823" exitCode=137 Nov 03 20:18:47.464265 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:47.464056 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-8c1590" Nov 03 20:18:47.464265 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:47.464119 4918 scope.go:122] "RemoveContainer" containerID="f9fdaefac79529b2d4457c98af2b533e68c233598e40575d2d5f7f7b9ff4f823" Nov 03 20:18:47.474487 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:47.474470 4918 scope.go:122] "RemoveContainer" containerID="f9fdaefac79529b2d4457c98af2b533e68c233598e40575d2d5f7f7b9ff4f823" Nov 03 20:18:47.474755 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:18:47.474727 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9fdaefac79529b2d4457c98af2b533e68c233598e40575d2d5f7f7b9ff4f823\": container with ID starting with f9fdaefac79529b2d4457c98af2b533e68c233598e40575d2d5f7f7b9ff4f823 not found: ID does not exist" containerID="f9fdaefac79529b2d4457c98af2b533e68c233598e40575d2d5f7f7b9ff4f823" Nov 03 20:18:47.474854 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:47.474758 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9fdaefac79529b2d4457c98af2b533e68c233598e40575d2d5f7f7b9ff4f823"} err="failed to get container status \"f9fdaefac79529b2d4457c98af2b533e68c233598e40575d2d5f7f7b9ff4f823\": rpc error: code = NotFound desc = could not find container \"f9fdaefac79529b2d4457c98af2b533e68c233598e40575d2d5f7f7b9ff4f823\": container with ID starting with f9fdaefac79529b2d4457c98af2b533e68c233598e40575d2d5f7f7b9ff4f823 not found: ID does not exist" Nov 03 20:18:50.044215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.044162 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr"] Nov 03 20:18:50.044951 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.044934 4918 cpu_manager.go:414] "containerMap: removing container" podUID="68d5c814-f1b6-4101-b5a4-6ab08d335af0" containerName="test-curl-7084-8c1590" Nov 03 20:18:50.045010 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.044952 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d5c814-f1b6-4101-b5a4-6ab08d335af0" containerName="test-curl-7084-8c1590" Nov 03 20:18:50.045085 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.045075 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d5c814-f1b6-4101-b5a4-6ab08d335af0" containerName="test-curl-7084-8c1590" Nov 03 20:18:50.050227 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.050163 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr" Nov 03 20:18:50.056136 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.056100 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7ea43c\"" Nov 03 20:18:50.056299 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.056198 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-7ea43c-dockercfg-whgkm\"" Nov 03 20:18:50.057356 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.057334 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms"] Nov 03 20:18:50.066618 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.066596 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr"] Nov 03 20:18:50.066733 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.066704 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms" Nov 03 20:18:50.068658 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.068636 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms"] Nov 03 20:18:50.191205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.190783 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxnrs\" (UniqueName: \"kubernetes.io/projected/bd8252d2-fbcf-454c-8b78-f3d16e38d0b4-kube-api-access-gxnrs\") pod \"payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms\" (UID: \"bd8252d2-fbcf-454c-8b78-f3d16e38d0b4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms" Nov 03 20:18:50.191205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.190895 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8208c2eb-0df0-42f5-a610-745cb7ca0d43-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr\" (UID: \"8208c2eb-0df0-42f5-a610-745cb7ca0d43\") " pod="openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr" Nov 03 20:18:50.191205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.190962 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnp75\" (UniqueName: \"kubernetes.io/projected/8208c2eb-0df0-42f5-a610-745cb7ca0d43-kube-api-access-cnp75\") pod \"payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr\" (UID: \"8208c2eb-0df0-42f5-a610-745cb7ca0d43\") " pod="openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr" Nov 03 20:18:50.191205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.191125 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8252d2-fbcf-454c-8b78-f3d16e38d0b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms\" (UID: \"bd8252d2-fbcf-454c-8b78-f3d16e38d0b4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms" Nov 03 20:18:50.363954 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.363906 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxnrs\" (UniqueName: \"kubernetes.io/projected/bd8252d2-fbcf-454c-8b78-f3d16e38d0b4-kube-api-access-gxnrs\") pod \"payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms\" (UID: \"bd8252d2-fbcf-454c-8b78-f3d16e38d0b4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms" Nov 03 20:18:50.377408 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.377368 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8208c2eb-0df0-42f5-a610-745cb7ca0d43-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr\" (UID: \"8208c2eb-0df0-42f5-a610-745cb7ca0d43\") " pod="openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr" Nov 03 20:18:50.378635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.378610 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnp75\" (UniqueName: \"kubernetes.io/projected/8208c2eb-0df0-42f5-a610-745cb7ca0d43-kube-api-access-cnp75\") pod \"payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr\" (UID: \"8208c2eb-0df0-42f5-a610-745cb7ca0d43\") " pod="openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr" Nov 03 20:18:50.379028 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.379006 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8252d2-fbcf-454c-8b78-f3d16e38d0b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms\" (UID: \"bd8252d2-fbcf-454c-8b78-f3d16e38d0b4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms" Nov 03 20:18:50.379676 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.378499 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8208c2eb-0df0-42f5-a610-745cb7ca0d43-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr\" (UID: \"8208c2eb-0df0-42f5-a610-745cb7ca0d43\") " pod="openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr" Nov 03 20:18:50.381240 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.380857 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8252d2-fbcf-454c-8b78-f3d16e38d0b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms\" (UID: \"bd8252d2-fbcf-454c-8b78-f3d16e38d0b4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms" Nov 03 20:18:50.401399 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.401365 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnp75\" (UniqueName: \"kubernetes.io/projected/8208c2eb-0df0-42f5-a610-745cb7ca0d43-kube-api-access-cnp75\") pod \"payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr\" (UID: \"8208c2eb-0df0-42f5-a610-745cb7ca0d43\") " pod="openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr" Nov 03 20:18:50.418824 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.415842 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxnrs\" (UniqueName: \"kubernetes.io/projected/bd8252d2-fbcf-454c-8b78-f3d16e38d0b4-kube-api-access-gxnrs\") pod \"payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms\" (UID: \"bd8252d2-fbcf-454c-8b78-f3d16e38d0b4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms" Nov 03 20:18:50.679339 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.678692 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr" Nov 03 20:18:50.684260 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:50.684230 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms" Nov 03 20:18:51.000818 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:18:51.000785 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8208c2eb_0df0_42f5_a610_745cb7ca0d43.slice/crio-83fc32512c57e36df6b24da7f388a2c3d419ab30e66fe789e8b3538ffc33bed7 WatchSource:0}: Error finding container 83fc32512c57e36df6b24da7f388a2c3d419ab30e66fe789e8b3538ffc33bed7: Status 404 returned error can't find the container with id 83fc32512c57e36df6b24da7f388a2c3d419ab30e66fe789e8b3538ffc33bed7 Nov 03 20:18:51.003840 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:51.003806 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr"] Nov 03 20:18:51.018761 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:51.018738 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms"] Nov 03 20:18:51.019637 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:18:51.019600 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8252d2_fbcf_454c_8b78_f3d16e38d0b4.slice/crio-da41e54b2932253a2c7cc1014ed6b7ecf343c66f950ef503003455eae01a8d48 WatchSource:0}: Error finding container da41e54b2932253a2c7cc1014ed6b7ecf343c66f950ef503003455eae01a8d48: Status 404 returned error can't find the container with id da41e54b2932253a2c7cc1014ed6b7ecf343c66f950ef503003455eae01a8d48 Nov 03 20:18:51.559478 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:51.559442 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr" event={"ID":"8208c2eb-0df0-42f5-a610-745cb7ca0d43","Type":"ContainerStarted","Data":"e0069fdfaf63aa5cd7949b091b6815007c31a652d1b693b3ce1d9054ff2017b4"} Nov 03 20:18:51.560029 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:51.560007 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr" Nov 03 20:18:51.560165 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:51.560150 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr" event={"ID":"8208c2eb-0df0-42f5-a610-745cb7ca0d43","Type":"ContainerStarted","Data":"83fc32512c57e36df6b24da7f388a2c3d419ab30e66fe789e8b3538ffc33bed7"} Nov 03 20:18:51.561935 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:51.561909 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms" event={"ID":"bd8252d2-fbcf-454c-8b78-f3d16e38d0b4","Type":"ContainerStarted","Data":"3077fa42793dd8fb92323e145580249ad19bc52e8088a0f0ae7385db72dd6643"} Nov 03 20:18:51.562042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:51.561955 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms" event={"ID":"bd8252d2-fbcf-454c-8b78-f3d16e38d0b4","Type":"ContainerStarted","Data":"da41e54b2932253a2c7cc1014ed6b7ecf343c66f950ef503003455eae01a8d48"} Nov 03 20:18:51.562874 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:51.562852 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms" Nov 03 20:18:51.581823 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:51.581772 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr" podStartSLOduration=1.581757993 podStartE2EDuration="1.581757993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:51.578147659 +0000 UTC m=+1314.510015171" watchObservedRunningTime="2026-11-03 20:18:51.581757993 +0000 UTC m=+1314.513625503" Nov 03 20:18:51.600688 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:18:51.600616 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms" podStartSLOduration=1.600599779 podStartE2EDuration="1.600599779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:51.598360569 +0000 UTC m=+1314.530228093" watchObservedRunningTime="2026-11-03 20:18:51.600599779 +0000 UTC m=+1314.532467298" Nov 03 20:18:58.224663 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:18:58.224621 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:19:03.585743 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:03.585710 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms" Nov 03 20:19:03.586422 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:03.586402 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr" Nov 03 20:19:05.063211 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:05.053291 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8bvn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:19:05.063211 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:05.053364 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" podUID="ea819fe0-461b-4952-a40e-7a47e8f51580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:16.290436 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.290401 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8"] Nov 03 20:19:16.295245 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.295219 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:16.310684 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.310655 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8"] Nov 03 20:19:16.314924 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.314898 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-15e02e6b\"/\"kube-root-ca.crt\"" Nov 03 20:19:16.316054 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.316029 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-15e02e6b\"/\"openshift-service-ca.crt\"" Nov 03 20:19:16.316054 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.316016 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-15e02e6b\"/\"default-dockercfg-f8qjh\"" Nov 03 20:19:16.316231 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.316134 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-15e02e6b\"/\"e2e-lifecycle-model-15e02e6b-kserve-self-signed-certs\"" Nov 03 20:19:16.394096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.394054 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-tmp-dir\") pod \"e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:16.394309 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.394111 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-dshm\") pod \"e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:16.394359 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.394331 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-home\") pod \"e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:16.394403 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.394360 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-model-cache\") pod \"e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:16.394459 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.394431 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-tls-certs\") pod \"e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:16.394577 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.394558 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m2lq\" (UniqueName: \"kubernetes.io/projected/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-kube-api-access-7m2lq\") pod \"e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:16.498200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.497575 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-home\") pod \"e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:16.498200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.497632 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-model-cache\") pod \"e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:16.498200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.497695 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-tls-certs\") pod \"e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:16.498200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.497792 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m2lq\" (UniqueName: \"kubernetes.io/projected/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-kube-api-access-7m2lq\") pod \"e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:16.498200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.497991 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-tmp-dir\") pod \"e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:16.498200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.498042 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-dshm\") pod \"e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:16.500207 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.499611 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-tmp-dir\") pod \"e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:16.500207 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.499879 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-home\") pod \"e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:16.500207 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.500137 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-model-cache\") pod \"e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:16.517716 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.503038 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-tls-certs\") pod \"e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:16.517716 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.504504 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-dshm\") pod \"e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:16.533855 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.529984 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m2lq\" (UniqueName: \"kubernetes.io/projected/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-kube-api-access-7m2lq\") pod \"e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:16.608983 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.608953 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:16.913392 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:16.905524 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8"] Nov 03 20:19:16.930340 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:19:16.930200 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7b8c27b_baaa_4eb1_89b7_263cc25d7a08.slice/crio-7d5ab9d05f1c72ef0de817d433eb2a2af4e5b5bcc49a065ac09c0c9f9fbd6787 WatchSource:0}: Error finding container 7d5ab9d05f1c72ef0de817d433eb2a2af4e5b5bcc49a065ac09c0c9f9fbd6787: Status 404 returned error can't find the container with id 7d5ab9d05f1c72ef0de817d433eb2a2af4e5b5bcc49a065ac09c0c9f9fbd6787 Nov 03 20:19:17.106881 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:17.106841 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" event={"ID":"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08","Type":"ContainerStarted","Data":"0c26b5088514a3476bed3e3fc29d9448300421fbcfbc03624f87f7211d3a6efb"} Nov 03 20:19:17.107059 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:17.106895 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" event={"ID":"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08","Type":"ContainerStarted","Data":"7d5ab9d05f1c72ef0de817d433eb2a2af4e5b5bcc49a065ac09c0c9f9fbd6787"} Nov 03 20:19:17.127212 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:17.127123 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" podStartSLOduration=1.127108553 podStartE2EDuration="1.127108553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:17.125428495 +0000 UTC m=+1340.057296017" watchObservedRunningTime="2026-11-03 20:19:17.127108553 +0000 UTC m=+1340.058976063" Nov 03 20:19:17.609691 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:17.609654 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:20.310025 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:20.309992 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc"] Nov 03 20:19:20.314937 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:20.314917 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc" Nov 03 20:19:20.324514 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:20.324452 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc"] Nov 03 20:19:20.381206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:20.376625 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9af0cd-dockercfg-bppt7\"" Nov 03 20:19:20.381206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:20.376766 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9af0cd\"" Nov 03 20:19:20.416204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:20.405947 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vppss\" (UniqueName: \"kubernetes.io/projected/0c6c5c52-c436-40c7-86f8-0af529511e48-kube-api-access-vppss\") pod \"payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc\" (UID: \"0c6c5c52-c436-40c7-86f8-0af529511e48\") " pod="openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc" Nov 03 20:19:20.416204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:20.406048 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c6c5c52-c436-40c7-86f8-0af529511e48-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc\" (UID: \"0c6c5c52-c436-40c7-86f8-0af529511e48\") " pod="openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc" Nov 03 20:19:20.517465 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:20.517403 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vppss\" (UniqueName: \"kubernetes.io/projected/0c6c5c52-c436-40c7-86f8-0af529511e48-kube-api-access-vppss\") pod \"payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc\" (UID: \"0c6c5c52-c436-40c7-86f8-0af529511e48\") " pod="openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc" Nov 03 20:19:20.541205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:20.519289 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c6c5c52-c436-40c7-86f8-0af529511e48-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc\" (UID: \"0c6c5c52-c436-40c7-86f8-0af529511e48\") " pod="openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc" Nov 03 20:19:20.541205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:20.520429 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c6c5c52-c436-40c7-86f8-0af529511e48-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc\" (UID: \"0c6c5c52-c436-40c7-86f8-0af529511e48\") " pod="openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc" Nov 03 20:19:20.543074 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:20.543040 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vppss\" (UniqueName: \"kubernetes.io/projected/0c6c5c52-c436-40c7-86f8-0af529511e48-kube-api-access-vppss\") pod \"payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc\" (UID: \"0c6c5c52-c436-40c7-86f8-0af529511e48\") " pod="openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc" Nov 03 20:19:20.631532 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:20.631488 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc" Nov 03 20:19:20.929916 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:20.929827 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb"] Nov 03 20:19:20.948522 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:20.948467 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:20.980010 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:20.979971 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-3ef4b9-openshift-default-dockercfg-8kj9n\"" Nov 03 20:19:21.000348 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:20.997742 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-workload-certs\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.000348 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:20.997862 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8f2527-d745-4a36-b656-594d782b0db3-istiod-ca-cert\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.000348 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:20.997912 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zvw4\" (UniqueName: \"kubernetes.io/projected/2f8f2527-d745-4a36-b656-594d782b0db3-kube-api-access-6zvw4\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.000348 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:20.997946 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-credential-socket\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.000348 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:20.998143 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8f2527-d745-4a36-b656-594d782b0db3-istio-token\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.060208 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.060139 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8f2527-d745-4a36-b656-594d782b0db3-istio-podinfo\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.060464 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.060443 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-istio-data\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.060580 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.060566 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-istio-envoy\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.060667 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.060655 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-workload-socket\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.088749 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.076098 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb"] Nov 03 20:19:21.108788 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.108687 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc"] Nov 03 20:19:21.167451 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.167408 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8f2527-d745-4a36-b656-594d782b0db3-istio-token\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.167752 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.167724 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8f2527-d745-4a36-b656-594d782b0db3-istio-podinfo\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.167922 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.167904 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-istio-data\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.168060 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.168043 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-istio-envoy\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.168200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.168165 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-workload-socket\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.168378 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.168360 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-workload-certs\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.168599 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.168574 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8f2527-d745-4a36-b656-594d782b0db3-istiod-ca-cert\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.168764 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.168746 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zvw4\" (UniqueName: \"kubernetes.io/projected/2f8f2527-d745-4a36-b656-594d782b0db3-kube-api-access-6zvw4\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.168873 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.168859 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-credential-socket\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.169728 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.169706 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-credential-socket\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.170455 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.170433 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-workload-socket\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.174019 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.172659 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-istio-data\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.174156 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.172912 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-workload-certs\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.174271 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.173956 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8f2527-d745-4a36-b656-594d782b0db3-istiod-ca-cert\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.187721 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.187689 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zvw4\" (UniqueName: \"kubernetes.io/projected/2f8f2527-d745-4a36-b656-594d782b0db3-kube-api-access-6zvw4\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.190499 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.190468 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8f2527-d745-4a36-b656-594d782b0db3-istio-token\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.216241 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.216157 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8f2527-d745-4a36-b656-594d782b0db3-istio-podinfo\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.244538 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.244508 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-istio-envoy\") pod \"e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.532684 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.532583 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:21.831585 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:19:21.831544 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8f2527_d745_4a36_b656_594d782b0db3.slice/crio-408dc6d2e17028e59584851cdcc06eb8ac6fa9ff79bd4efce648caafb36898a4 WatchSource:0}: Error finding container 408dc6d2e17028e59584851cdcc06eb8ac6fa9ff79bd4efce648caafb36898a4: Status 404 returned error can't find the container with id 408dc6d2e17028e59584851cdcc06eb8ac6fa9ff79bd4efce648caafb36898a4 Nov 03 20:19:21.833052 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.833020 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb"] Nov 03 20:19:21.835433 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.835391 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:19:21.835525 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.835481 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:19:21.835565 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:21.835522 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:19:22.226970 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:22.226498 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc" event={"ID":"0c6c5c52-c436-40c7-86f8-0af529511e48","Type":"ContainerStarted","Data":"3fb3195af16fadf99e5211bf72ae27eb2e28bd59c209e039e4d126a25bf36d13"} Nov 03 20:19:22.226970 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:22.226573 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc" event={"ID":"0c6c5c52-c436-40c7-86f8-0af529511e48","Type":"ContainerStarted","Data":"d4554fe15f9e3b020e7b66f619a564ef1dc143d543512d2d50c935d66cd5f21f"} Nov 03 20:19:22.226970 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:22.226810 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc" Nov 03 20:19:22.228758 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:22.228727 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" event={"ID":"2f8f2527-d745-4a36-b656-594d782b0db3","Type":"ContainerStarted","Data":"3b9d321a30241bc19fdee48a378432cead8677df92bad7a99e0cef03a356fcb1"} Nov 03 20:19:22.228889 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:22.228783 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" event={"ID":"2f8f2527-d745-4a36-b656-594d782b0db3","Type":"ContainerStarted","Data":"408dc6d2e17028e59584851cdcc06eb8ac6fa9ff79bd4efce648caafb36898a4"} Nov 03 20:19:22.249889 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:22.249827 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc" podStartSLOduration=2.249813017 podStartE2EDuration="2.249813017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:22.245848347 +0000 UTC m=+1345.177715859" watchObservedRunningTime="2026-11-03 20:19:22.249813017 +0000 UTC m=+1345.181680528" Nov 03 20:19:22.272816 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:22.272752 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" podStartSLOduration=2.27273224 podStartE2EDuration="2.27273224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:22.268606746 +0000 UTC m=+1345.200474279" watchObservedRunningTime="2026-11-03 20:19:22.27273224 +0000 UTC m=+1345.204599753" Nov 03 20:19:22.544272 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:22.541141 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:22.586531 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:22.586495 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:23.236847 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:23.236807 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:23.237941 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:23.237916 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:19:25.845509 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:25.845472 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw"] Nov 03 20:19:25.852656 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:25.852630 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw" Nov 03 20:19:25.860467 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:25.860431 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw"] Nov 03 20:19:25.861097 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:25.861074 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-3ef4b9\"" Nov 03 20:19:25.863928 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:25.863901 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-3ef4b9-dockercfg-jcmkq\"" Nov 03 20:19:25.905456 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:25.905416 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a96cad21-e85e-4757-9e72-6b350738e230-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw\" (UID: \"a96cad21-e85e-4757-9e72-6b350738e230\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw" Nov 03 20:19:25.905690 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:25.905668 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txlf6\" (UniqueName: \"kubernetes.io/projected/a96cad21-e85e-4757-9e72-6b350738e230-kube-api-access-txlf6\") pod \"payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw\" (UID: \"a96cad21-e85e-4757-9e72-6b350738e230\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw" Nov 03 20:19:26.027200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:26.016396 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a96cad21-e85e-4757-9e72-6b350738e230-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw\" (UID: \"a96cad21-e85e-4757-9e72-6b350738e230\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw" Nov 03 20:19:26.027200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:26.016606 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txlf6\" (UniqueName: \"kubernetes.io/projected/a96cad21-e85e-4757-9e72-6b350738e230-kube-api-access-txlf6\") pod \"payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw\" (UID: \"a96cad21-e85e-4757-9e72-6b350738e230\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw" Nov 03 20:19:26.045200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:26.044255 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a96cad21-e85e-4757-9e72-6b350738e230-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw\" (UID: \"a96cad21-e85e-4757-9e72-6b350738e230\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw" Nov 03 20:19:26.092061 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:26.091947 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txlf6\" (UniqueName: \"kubernetes.io/projected/a96cad21-e85e-4757-9e72-6b350738e230-kube-api-access-txlf6\") pod \"payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw\" (UID: \"a96cad21-e85e-4757-9e72-6b350738e230\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw" Nov 03 20:19:26.181254 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:26.174627 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw" Nov 03 20:19:26.566308 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:26.566280 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw"] Nov 03 20:19:26.567165 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:19:26.567135 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96cad21_e85e_4757_9e72_6b350738e230.slice/crio-9e917e6ab4db4234f16e56e9942c5bfca99c08942478a40cfc2851f3806fd749 WatchSource:0}: Error finding container 9e917e6ab4db4234f16e56e9942c5bfca99c08942478a40cfc2851f3806fd749: Status 404 returned error can't find the container with id 9e917e6ab4db4234f16e56e9942c5bfca99c08942478a40cfc2851f3806fd749 Nov 03 20:19:26.610047 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:26.610023 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:26.623337 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:26.623309 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:27.278624 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:27.278588 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw" event={"ID":"a96cad21-e85e-4757-9e72-6b350738e230","Type":"ContainerStarted","Data":"9016f16a3d639f1ea2a3a88dac11b95115f749cc37af709ccdc679e35aad4bc4"} Nov 03 20:19:27.279046 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:27.278652 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw" event={"ID":"a96cad21-e85e-4757-9e72-6b350738e230","Type":"ContainerStarted","Data":"9e917e6ab4db4234f16e56e9942c5bfca99c08942478a40cfc2851f3806fd749"} Nov 03 20:19:27.279046 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:27.278922 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw" Nov 03 20:19:27.290056 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:27.290032 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:19:27.300490 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:27.300449 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw" podStartSLOduration=2.300436145 podStartE2EDuration="2.300436145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:27.297319941 +0000 UTC m=+1350.229187482" watchObservedRunningTime="2026-11-03 20:19:27.300436145 +0000 UTC m=+1350.232303653" Nov 03 20:19:28.210058 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:19:28.210023 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:19:33.241416 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:33.241386 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc" Nov 03 20:19:36.886944 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:36.886911 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d"] Nov 03 20:19:36.891862 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:36.891835 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d" Nov 03 20:19:36.908124 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:36.907120 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d"] Nov 03 20:19:36.912864 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:36.912835 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-06294c6a-dockercfg-6q7k2\"" Nov 03 20:19:37.008199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:36.992639 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-06294c6a\"" Nov 03 20:19:37.049200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:37.042127 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b6760f6-b0a4-4549-a135-e3643d29e2e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d\" (UID: \"9b6760f6-b0a4-4549-a135-e3643d29e2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d" Nov 03 20:19:37.049200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:37.042205 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpttk\" (UniqueName: \"kubernetes.io/projected/9b6760f6-b0a4-4549-a135-e3643d29e2e6-kube-api-access-mpttk\") pod \"payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d\" (UID: \"9b6760f6-b0a4-4549-a135-e3643d29e2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d" Nov 03 20:19:37.145434 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:37.144490 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b6760f6-b0a4-4549-a135-e3643d29e2e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d\" (UID: \"9b6760f6-b0a4-4549-a135-e3643d29e2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d" Nov 03 20:19:37.145673 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:37.145650 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpttk\" (UniqueName: \"kubernetes.io/projected/9b6760f6-b0a4-4549-a135-e3643d29e2e6-kube-api-access-mpttk\") pod \"payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d\" (UID: \"9b6760f6-b0a4-4549-a135-e3643d29e2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d" Nov 03 20:19:37.146447 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:37.145356 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b6760f6-b0a4-4549-a135-e3643d29e2e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d\" (UID: \"9b6760f6-b0a4-4549-a135-e3643d29e2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d" Nov 03 20:19:37.209049 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:37.209015 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpttk\" (UniqueName: \"kubernetes.io/projected/9b6760f6-b0a4-4549-a135-e3643d29e2e6-kube-api-access-mpttk\") pod \"payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d\" (UID: \"9b6760f6-b0a4-4549-a135-e3643d29e2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d" Nov 03 20:19:37.232953 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:37.232918 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d" Nov 03 20:19:37.563986 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:37.563935 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d"] Nov 03 20:19:37.569155 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:19:37.569122 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b6760f6_b0a4_4549_a135_e3643d29e2e6.slice/crio-111009de49e4f8554d953c5e2ea015a178db0ba638f6339f2bd4b957ad009939 WatchSource:0}: Error finding container 111009de49e4f8554d953c5e2ea015a178db0ba638f6339f2bd4b957ad009939: Status 404 returned error can't find the container with id 111009de49e4f8554d953c5e2ea015a178db0ba638f6339f2bd4b957ad009939 Nov 03 20:19:38.289046 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:38.289018 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw" Nov 03 20:19:38.384129 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:38.384090 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d" event={"ID":"9b6760f6-b0a4-4549-a135-e3643d29e2e6","Type":"ContainerStarted","Data":"a063aedc2822e4134cd0768a8bfc9c2e2df19e6a3eae615799d94d49e181d2be"} Nov 03 20:19:38.384129 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:38.384157 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d" event={"ID":"9b6760f6-b0a4-4549-a135-e3643d29e2e6","Type":"ContainerStarted","Data":"111009de49e4f8554d953c5e2ea015a178db0ba638f6339f2bd4b957ad009939"} Nov 03 20:19:38.384129 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:38.384238 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d" Nov 03 20:19:38.407399 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:38.407346 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d" podStartSLOduration=2.407331225 podStartE2EDuration="2.407331225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:38.403147226 +0000 UTC m=+1361.335014736" watchObservedRunningTime="2026-11-03 20:19:38.407331225 +0000 UTC m=+1361.339198736" Nov 03 20:19:49.394161 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:19:49.394092 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d" Nov 03 20:19:58.235552 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:19:58.233240 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:20:02.924069 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:02.924023 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7"] Nov 03 20:20:02.942885 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:02.942849 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:02.950892 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:02.950861 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7"] Nov 03 20:20:02.957270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:02.954922 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-ba8f5b-openshift-default-dockercfg-r8xc2\"" Nov 03 20:20:03.045218 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.045139 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-envoy\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.045399 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.045252 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-data\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.045445 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.045411 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-credential-socket\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.045515 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.045499 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-workload-socket\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.045609 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.045596 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istiod-ca-cert\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.045732 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.045711 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-podinfo\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.045787 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.045773 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-token\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.045823 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.045800 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgz9p\" (UniqueName: \"kubernetes.io/projected/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-kube-api-access-cgz9p\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.045857 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.045832 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-workload-certs\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.165199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.164880 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istiod-ca-cert\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.165199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.165033 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-podinfo\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.165199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.165091 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-token\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.165199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.165123 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgz9p\" (UniqueName: \"kubernetes.io/projected/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-kube-api-access-cgz9p\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.165199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.165199 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-workload-certs\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.165620 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.165315 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-envoy\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.165620 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.165363 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-data\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.165732 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.165622 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-credential-socket\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.169198 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.165848 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istiod-ca-cert\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.169198 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.167337 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-workload-socket\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.169198 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.167966 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-workload-socket\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.179282 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.173638 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-workload-certs\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.179282 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.174361 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-data\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.179282 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.178821 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-podinfo\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.180409 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.180380 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-envoy\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.185948 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.183939 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-credential-socket\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.192909 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.192536 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-token\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.192909 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.192858 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgz9p\" (UniqueName: \"kubernetes.io/projected/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-kube-api-access-cgz9p\") pod \"e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.272839 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.272803 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:03.586901 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.586870 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7"] Nov 03 20:20:03.587520 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:20:03.587485 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d98bfc5_35c3_4925_a3d1_696dd0f90dd7.slice/crio-ee3b3557fc99b35ccbe0e30baf3e68fcd58ec9e1c49dc8e050325bef95e685d8 WatchSource:0}: Error finding container ee3b3557fc99b35ccbe0e30baf3e68fcd58ec9e1c49dc8e050325bef95e685d8: Status 404 returned error can't find the container with id ee3b3557fc99b35ccbe0e30baf3e68fcd58ec9e1c49dc8e050325bef95e685d8 Nov 03 20:20:03.591011 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.590970 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:20:03.591089 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.591066 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:20:03.591131 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.591110 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:20:03.888331 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.888226 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" event={"ID":"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7","Type":"ContainerStarted","Data":"7320646a9c45f1fbc4bae0aed37c5df44d18947d80ea929fc253520e10799ba2"} Nov 03 20:20:03.888331 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.888304 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" event={"ID":"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7","Type":"ContainerStarted","Data":"ee3b3557fc99b35ccbe0e30baf3e68fcd58ec9e1c49dc8e050325bef95e685d8"} Nov 03 20:20:03.913766 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:03.913715 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" podStartSLOduration=1.913700014 podStartE2EDuration="1.913700014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:03.912505224 +0000 UTC m=+1386.844372750" watchObservedRunningTime="2026-11-03 20:20:03.913700014 +0000 UTC m=+1386.845567523" Nov 03 20:20:04.273589 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:04.273510 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:04.279319 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:04.279287 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:04.895787 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:04.895756 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:04.896882 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:04.896858 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:20:09.027525 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:09.027439 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9"] Nov 03 20:20:09.032950 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:09.032915 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc99981c-kqrz9"] Nov 03 20:20:09.218056 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:09.218020 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce" path="/var/lib/kubelet/pods/28a0d3ab-cf16-4f2c-bb86-fb2e8bfb28ce/volumes" Nov 03 20:20:20.749081 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:20.748962 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z"] Nov 03 20:20:21.077210 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:21.076954 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z"] Nov 03 20:20:21.077210 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:21.077088 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" Nov 03 20:20:21.092867 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:21.092671 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-ba8f5b-serving-cert\"" Nov 03 20:20:21.162487 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:21.159032 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d986da59-41d5-4c2e-8832-e7e79e5e2b19-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z\" (UID: \"d986da59-41d5-4c2e-8832-e7e79e5e2b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" Nov 03 20:20:21.162487 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:21.159163 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d986da59-41d5-4c2e-8832-e7e79e5e2b19-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z\" (UID: \"d986da59-41d5-4c2e-8832-e7e79e5e2b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" Nov 03 20:20:21.162487 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:21.159270 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8vrs\" (UniqueName: \"kubernetes.io/projected/d986da59-41d5-4c2e-8832-e7e79e5e2b19-kube-api-access-k8vrs\") pod \"maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z\" (UID: \"d986da59-41d5-4c2e-8832-e7e79e5e2b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" Nov 03 20:20:21.162487 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:21.159451 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d986da59-41d5-4c2e-8832-e7e79e5e2b19-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z\" (UID: \"d986da59-41d5-4c2e-8832-e7e79e5e2b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" Nov 03 20:20:21.262221 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:21.261159 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d986da59-41d5-4c2e-8832-e7e79e5e2b19-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z\" (UID: \"d986da59-41d5-4c2e-8832-e7e79e5e2b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" Nov 03 20:20:21.262221 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:21.261472 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d986da59-41d5-4c2e-8832-e7e79e5e2b19-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z\" (UID: \"d986da59-41d5-4c2e-8832-e7e79e5e2b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" Nov 03 20:20:21.262221 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:21.261543 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d986da59-41d5-4c2e-8832-e7e79e5e2b19-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z\" (UID: \"d986da59-41d5-4c2e-8832-e7e79e5e2b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" Nov 03 20:20:21.262221 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:21.261577 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vrs\" (UniqueName: \"kubernetes.io/projected/d986da59-41d5-4c2e-8832-e7e79e5e2b19-kube-api-access-k8vrs\") pod \"maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z\" (UID: \"d986da59-41d5-4c2e-8832-e7e79e5e2b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" Nov 03 20:20:21.264324 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:21.264048 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d986da59-41d5-4c2e-8832-e7e79e5e2b19-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z\" (UID: \"d986da59-41d5-4c2e-8832-e7e79e5e2b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" Nov 03 20:20:21.267341 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:21.265264 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d986da59-41d5-4c2e-8832-e7e79e5e2b19-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z\" (UID: \"d986da59-41d5-4c2e-8832-e7e79e5e2b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" Nov 03 20:20:21.267341 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:21.265752 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d986da59-41d5-4c2e-8832-e7e79e5e2b19-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z\" (UID: \"d986da59-41d5-4c2e-8832-e7e79e5e2b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" Nov 03 20:20:21.279493 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:21.279465 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8vrs\" (UniqueName: \"kubernetes.io/projected/d986da59-41d5-4c2e-8832-e7e79e5e2b19-kube-api-access-k8vrs\") pod \"maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z\" (UID: \"d986da59-41d5-4c2e-8832-e7e79e5e2b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" Nov 03 20:20:21.455258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:21.448549 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" Nov 03 20:20:22.054209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:22.053986 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z"] Nov 03 20:20:22.058682 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:22.058655 4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:20:22.292277 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:22.292064 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" event={"ID":"d986da59-41d5-4c2e-8832-e7e79e5e2b19","Type":"ContainerStarted","Data":"6979394d3887bf3eb79843f06afdaa8322865f84fc541ee429ee346619e6abd5"} Nov 03 20:20:23.301258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:23.301221 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" event={"ID":"d986da59-41d5-4c2e-8832-e7e79e5e2b19","Type":"ContainerStarted","Data":"4fd3f8bd2a2d34539b9840dc1a84c0c5209ad1881c138f96a14362766e1d520a"} Nov 03 20:20:23.301672 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:23.301463 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" Nov 03 20:20:23.348947 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:23.348876 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" podStartSLOduration=2.804686952 podStartE2EDuration="3.348854144s" totalImagesPullingTime="544.167192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:23.343477222 +0000 UTC m=+1406.275344757" watchObservedRunningTime="2026-11-03 20:20:23.348854144 +0000 UTC m=+1406.280721658" Nov 03 20:20:23.610441 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:23.610400 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr"] Nov 03 20:20:23.664301 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:23.661981 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms"] Nov 03 20:20:28.218244 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:20:28.218205 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:20:28.611325 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:28.611276 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr" podUID="8208c2eb-0df0-42f5-a610-745cb7ca0d43" containerName="payload-processing" containerID="cri-o://e0069fdfaf63aa5cd7949b091b6815007c31a652d1b693b3ce1d9054ff2017b4" gracePeriod=25 Nov 03 20:20:28.662689 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:28.662642 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms" podUID="bd8252d2-fbcf-454c-8b78-f3d16e38d0b4" containerName="payload-pre-processing" containerID="cri-o://3077fa42793dd8fb92323e145580249ad19bc52e8088a0f0ae7385db72dd6643" gracePeriod=25 Nov 03 20:20:28.752752 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:20:28.752711 4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8252d2_fbcf_454c_8b78_f3d16e38d0b4.slice/crio-3077fa42793dd8fb92323e145580249ad19bc52e8088a0f0ae7385db72dd6643.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8252d2_fbcf_454c_8b78_f3d16e38d0b4.slice/crio-conmon-3077fa42793dd8fb92323e145580249ad19bc52e8088a0f0ae7385db72dd6643.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:20:29.117704 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.117671 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr" Nov 03 20:20:29.128163 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.128133 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms" Nov 03 20:20:29.185549 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.185458 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd8252d2-fbcf-454c-8b78-f3d16e38d0b4-kube-api-access-gxnrs\" (UniqueName: \"kubernetes.io/projected/bd8252d2-fbcf-454c-8b78-f3d16e38d0b4-kube-api-access-gxnrs\") pod \"bd8252d2-fbcf-454c-8b78-f3d16e38d0b4\" (UID: \"bd8252d2-fbcf-454c-8b78-f3d16e38d0b4\") " Nov 03 20:20:29.185818 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.185785 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8208c2eb-0df0-42f5-a610-745cb7ca0d43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8208c2eb-0df0-42f5-a610-745cb7ca0d43-plugins-config-volume\") pod \"8208c2eb-0df0-42f5-a610-745cb7ca0d43\" (UID: \"8208c2eb-0df0-42f5-a610-745cb7ca0d43\") " Nov 03 20:20:29.185887 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.185851 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8208c2eb-0df0-42f5-a610-745cb7ca0d43-kube-api-access-cnp75\" (UniqueName: \"kubernetes.io/projected/8208c2eb-0df0-42f5-a610-745cb7ca0d43-kube-api-access-cnp75\") pod \"8208c2eb-0df0-42f5-a610-745cb7ca0d43\" (UID: \"8208c2eb-0df0-42f5-a610-745cb7ca0d43\") " Nov 03 20:20:29.186137 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.186113 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd8252d2-fbcf-454c-8b78-f3d16e38d0b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8252d2-fbcf-454c-8b78-f3d16e38d0b4-plugins-config-volume\") pod \"bd8252d2-fbcf-454c-8b78-f3d16e38d0b4\" (UID: \"bd8252d2-fbcf-454c-8b78-f3d16e38d0b4\") " Nov 03 20:20:29.186765 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.186733 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8208c2eb-0df0-42f5-a610-745cb7ca0d43-plugins-config-volume" pod "8208c2eb-0df0-42f5-a610-745cb7ca0d43" (UID: "8208c2eb-0df0-42f5-a610-745cb7ca0d43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:29.187232 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.187209 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8252d2-fbcf-454c-8b78-f3d16e38d0b4-plugins-config-volume" pod "bd8252d2-fbcf-454c-8b78-f3d16e38d0b4" (UID: "bd8252d2-fbcf-454c-8b78-f3d16e38d0b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:29.187348 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.187253 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8208c2eb-0df0-42f5-a610-745cb7ca0d43-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:29.191826 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.191798 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8252d2-fbcf-454c-8b78-f3d16e38d0b4-kube-api-access-gxnrs" pod "bd8252d2-fbcf-454c-8b78-f3d16e38d0b4" (UID: "bd8252d2-fbcf-454c-8b78-f3d16e38d0b4"). InnerVolumeSpecName "kube-api-access-gxnrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:29.192003 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.191985 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8208c2eb-0df0-42f5-a610-745cb7ca0d43-kube-api-access-cnp75" pod "8208c2eb-0df0-42f5-a610-745cb7ca0d43" (UID: "8208c2eb-0df0-42f5-a610-745cb7ca0d43"). InnerVolumeSpecName "kube-api-access-cnp75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:29.289730 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.289685 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxnrs\" (UniqueName: \"kubernetes.io/projected/bd8252d2-fbcf-454c-8b78-f3d16e38d0b4-kube-api-access-gxnrs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:29.289730 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.289723 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnp75\" (UniqueName: \"kubernetes.io/projected/8208c2eb-0df0-42f5-a610-745cb7ca0d43-kube-api-access-cnp75\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:29.289730 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.289738 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8252d2-fbcf-454c-8b78-f3d16e38d0b4-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:29.317841 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.317812 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" Nov 03 20:20:29.417672 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.417633 4918 generic.go:356] "Generic (PLEG): container finished" podID="8208c2eb-0df0-42f5-a610-745cb7ca0d43" containerID="e0069fdfaf63aa5cd7949b091b6815007c31a652d1b693b3ce1d9054ff2017b4" exitCode=0 Nov 03 20:20:29.417672 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.417682 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr" Nov 03 20:20:29.418004 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.417745 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr" event={"ID":"8208c2eb-0df0-42f5-a610-745cb7ca0d43","Type":"ContainerDied","Data":"e0069fdfaf63aa5cd7949b091b6815007c31a652d1b693b3ce1d9054ff2017b4"} Nov 03 20:20:29.418004 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.417813 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr" event={"ID":"8208c2eb-0df0-42f5-a610-745cb7ca0d43","Type":"ContainerDied","Data":"83fc32512c57e36df6b24da7f388a2c3d419ab30e66fe789e8b3538ffc33bed7"} Nov 03 20:20:29.418004 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.417849 4918 scope.go:122] "RemoveContainer" containerID="e0069fdfaf63aa5cd7949b091b6815007c31a652d1b693b3ce1d9054ff2017b4" Nov 03 20:20:29.421158 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.420538 4918 generic.go:356] "Generic (PLEG): container finished" podID="bd8252d2-fbcf-454c-8b78-f3d16e38d0b4" containerID="3077fa42793dd8fb92323e145580249ad19bc52e8088a0f0ae7385db72dd6643" exitCode=0 Nov 03 20:20:29.421158 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.420594 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms" Nov 03 20:20:29.421158 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.420620 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms" event={"ID":"bd8252d2-fbcf-454c-8b78-f3d16e38d0b4","Type":"ContainerDied","Data":"3077fa42793dd8fb92323e145580249ad19bc52e8088a0f0ae7385db72dd6643"} Nov 03 20:20:29.421158 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.420666 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms" event={"ID":"bd8252d2-fbcf-454c-8b78-f3d16e38d0b4","Type":"ContainerDied","Data":"da41e54b2932253a2c7cc1014ed6b7ecf343c66f950ef503003455eae01a8d48"} Nov 03 20:20:29.433848 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.433827 4918 scope.go:122] "RemoveContainer" containerID="e0069fdfaf63aa5cd7949b091b6815007c31a652d1b693b3ce1d9054ff2017b4" Nov 03 20:20:29.434130 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:20:29.434111 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0069fdfaf63aa5cd7949b091b6815007c31a652d1b693b3ce1d9054ff2017b4\": container with ID starting with e0069fdfaf63aa5cd7949b091b6815007c31a652d1b693b3ce1d9054ff2017b4 not found: ID does not exist" containerID="e0069fdfaf63aa5cd7949b091b6815007c31a652d1b693b3ce1d9054ff2017b4" Nov 03 20:20:29.434251 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.434139 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0069fdfaf63aa5cd7949b091b6815007c31a652d1b693b3ce1d9054ff2017b4"} err="failed to get container status \"e0069fdfaf63aa5cd7949b091b6815007c31a652d1b693b3ce1d9054ff2017b4\": rpc error: code = NotFound desc = could not find container \"e0069fdfaf63aa5cd7949b091b6815007c31a652d1b693b3ce1d9054ff2017b4\": container with ID starting with e0069fdfaf63aa5cd7949b091b6815007c31a652d1b693b3ce1d9054ff2017b4 not found: ID does not exist" Nov 03 20:20:29.434251 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.434156 4918 scope.go:122] "RemoveContainer" containerID="3077fa42793dd8fb92323e145580249ad19bc52e8088a0f0ae7385db72dd6643" Nov 03 20:20:29.445779 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.445621 4918 scope.go:122] "RemoveContainer" containerID="3077fa42793dd8fb92323e145580249ad19bc52e8088a0f0ae7385db72dd6643" Nov 03 20:20:29.446614 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:20:29.446586 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3077fa42793dd8fb92323e145580249ad19bc52e8088a0f0ae7385db72dd6643\": container with ID starting with 3077fa42793dd8fb92323e145580249ad19bc52e8088a0f0ae7385db72dd6643 not found: ID does not exist" containerID="3077fa42793dd8fb92323e145580249ad19bc52e8088a0f0ae7385db72dd6643" Nov 03 20:20:29.446726 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.446623 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3077fa42793dd8fb92323e145580249ad19bc52e8088a0f0ae7385db72dd6643"} err="failed to get container status \"3077fa42793dd8fb92323e145580249ad19bc52e8088a0f0ae7385db72dd6643\": rpc error: code = NotFound desc = could not find container \"3077fa42793dd8fb92323e145580249ad19bc52e8088a0f0ae7385db72dd6643\": container with ID starting with 3077fa42793dd8fb92323e145580249ad19bc52e8088a0f0ae7385db72dd6643 not found: ID does not exist" Nov 03 20:20:29.447130 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.447109 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms"] Nov 03 20:20:29.456298 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.456268 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7ea43c-84bfd59bf9-26pms"] Nov 03 20:20:29.475085 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.475054 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr"] Nov 03 20:20:29.479610 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:29.479586 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7ea43c-76bb7fcd7b-jccjr"] Nov 03 20:20:31.217586 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:31.217481 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8208c2eb-0df0-42f5-a610-745cb7ca0d43" path="/var/lib/kubelet/pods/8208c2eb-0df0-42f5-a610-745cb7ca0d43/volumes" Nov 03 20:20:31.218455 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:31.218074 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd8252d2-fbcf-454c-8b78-f3d16e38d0b4" path="/var/lib/kubelet/pods/bd8252d2-fbcf-454c-8b78-f3d16e38d0b4/volumes" Nov 03 20:20:31.441559 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:31.441520 4918 generic.go:356] "Generic (PLEG): container finished" podID="2f8f2527-d745-4a36-b656-594d782b0db3" containerID="3b9d321a30241bc19fdee48a378432cead8677df92bad7a99e0cef03a356fcb1" exitCode=0 Nov 03 20:20:31.441730 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:31.441563 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" event={"ID":"2f8f2527-d745-4a36-b656-594d782b0db3","Type":"ContainerDied","Data":"3b9d321a30241bc19fdee48a378432cead8677df92bad7a99e0cef03a356fcb1"} Nov 03 20:20:31.442140 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:31.442115 4918 scope.go:122] "RemoveContainer" containerID="3b9d321a30241bc19fdee48a378432cead8677df92bad7a99e0cef03a356fcb1" Nov 03 20:20:31.444151 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:31.444114 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:20:31.444339 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:31.444214 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:20:31.444339 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:31.444254 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:20:32.452570 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:32.452533 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" event={"ID":"2f8f2527-d745-4a36-b656-594d782b0db3","Type":"ContainerStarted","Data":"00811c572e166823a0f507735808690d98923f3209146aad7f69a4cf90e7a4fc"} Nov 03 20:20:32.538201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:32.534614 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:20:32.539066 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:32.539029 4918 patch_prober.go:29] interesting pod/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 20:20:32.543499 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:32.539294 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" podUID="2f8f2527-d745-4a36-b656-594d782b0db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 20:20:33.538200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:33.537327 4918 patch_prober.go:29] interesting pod/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 20:20:33.538200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:33.537392 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" podUID="2f8f2527-d745-4a36-b656-594d782b0db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 20:20:34.551202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:34.551045 4918 patch_prober.go:29] interesting pod/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 20:20:34.551202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:34.551120 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" podUID="2f8f2527-d745-4a36-b656-594d782b0db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 20:20:35.542201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:35.541374 4918 patch_prober.go:29] interesting pod/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 20:20:35.542201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:35.541448 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" podUID="2f8f2527-d745-4a36-b656-594d782b0db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 20:20:36.539202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:36.537559 4918 patch_prober.go:29] interesting pod/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 20:20:36.539202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:36.537623 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" podUID="2f8f2527-d745-4a36-b656-594d782b0db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 20:20:37.547101 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:37.543970 4918 patch_prober.go:29] interesting pod/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 20:20:37.547101 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:37.544032 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" podUID="2f8f2527-d745-4a36-b656-594d782b0db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 20:20:38.237313 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:38.237278 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:20:38.538855 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:38.538760 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:20:38.580629 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:38.580597 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:20:45.248302 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:45.248270 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq"] Nov 03 20:20:45.249059 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:45.249043 4918 cpu_manager.go:414] "containerMap: removing container" podUID="bd8252d2-fbcf-454c-8b78-f3d16e38d0b4" containerName="payload-pre-processing" Nov 03 20:20:45.249104 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:45.249060 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8252d2-fbcf-454c-8b78-f3d16e38d0b4" containerName="payload-pre-processing" Nov 03 20:20:45.249104 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:45.249075 4918 cpu_manager.go:414] "containerMap: removing container" podUID="8208c2eb-0df0-42f5-a610-745cb7ca0d43" containerName="payload-processing" Nov 03 20:20:45.249104 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:45.249080 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8208c2eb-0df0-42f5-a610-745cb7ca0d43" containerName="payload-processing" Nov 03 20:20:45.249225 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:45.249214 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8252d2-fbcf-454c-8b78-f3d16e38d0b4" containerName="payload-pre-processing" Nov 03 20:20:45.249260 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:45.249228 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="8208c2eb-0df0-42f5-a610-745cb7ca0d43" containerName="payload-processing" Nov 03 20:20:45.253252 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:45.253233 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq" Nov 03 20:20:45.258589 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:45.258565 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2d5927\"" Nov 03 20:20:45.265809 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:45.265775 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq"] Nov 03 20:20:45.348350 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:45.348308 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ff8461-6882-4ad5-8d54-eb63951fb83c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq\" (UID: \"76ff8461-6882-4ad5-8d54-eb63951fb83c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq" Nov 03 20:20:45.348789 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:45.348756 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hhcw\" (UniqueName: \"kubernetes.io/projected/76ff8461-6882-4ad5-8d54-eb63951fb83c-kube-api-access-7hhcw\") pod \"payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq\" (UID: \"76ff8461-6882-4ad5-8d54-eb63951fb83c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq" Nov 03 20:20:45.454203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:45.453391 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ff8461-6882-4ad5-8d54-eb63951fb83c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq\" (UID: \"76ff8461-6882-4ad5-8d54-eb63951fb83c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq" Nov 03 20:20:45.454203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:45.453635 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hhcw\" (UniqueName: \"kubernetes.io/projected/76ff8461-6882-4ad5-8d54-eb63951fb83c-kube-api-access-7hhcw\") pod \"payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq\" (UID: \"76ff8461-6882-4ad5-8d54-eb63951fb83c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq" Nov 03 20:20:45.495201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:45.486150 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ff8461-6882-4ad5-8d54-eb63951fb83c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq\" (UID: \"76ff8461-6882-4ad5-8d54-eb63951fb83c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq" Nov 03 20:20:45.548856 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:45.548760 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hhcw\" (UniqueName: \"kubernetes.io/projected/76ff8461-6882-4ad5-8d54-eb63951fb83c-kube-api-access-7hhcw\") pod \"payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq\" (UID: \"76ff8461-6882-4ad5-8d54-eb63951fb83c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq" Nov 03 20:20:45.573196 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:45.570062 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq" Nov 03 20:20:45.914933 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:20:45.914876 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ff8461_6882_4ad5_8d54_eb63951fb83c.slice/crio-add566e6228a7e6b1ac70f6cea4c858efce6496cbb6f8ea41ce8f251d091987c WatchSource:0}: Error finding container add566e6228a7e6b1ac70f6cea4c858efce6496cbb6f8ea41ce8f251d091987c: Status 404 returned error can't find the container with id add566e6228a7e6b1ac70f6cea4c858efce6496cbb6f8ea41ce8f251d091987c Nov 03 20:20:45.915338 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:45.915312 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq"] Nov 03 20:20:46.670601 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:46.670561 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq" event={"ID":"76ff8461-6882-4ad5-8d54-eb63951fb83c","Type":"ContainerStarted","Data":"6ac53108b269a56401145bd5743b5efb7c460cbbef31a030e08ef33cdae12034"} Nov 03 20:20:46.671285 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:46.671254 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq" event={"ID":"76ff8461-6882-4ad5-8d54-eb63951fb83c","Type":"ContainerStarted","Data":"add566e6228a7e6b1ac70f6cea4c858efce6496cbb6f8ea41ce8f251d091987c"} Nov 03 20:20:46.671528 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:46.671514 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq" Nov 03 20:20:46.694770 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:46.694704 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq" podStartSLOduration=1.6946834389999998 podStartE2EDuration="1.694683439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:46.68988766 +0000 UTC m=+1429.621755171" watchObservedRunningTime="2026-11-03 20:20:46.694683439 +0000 UTC m=+1429.626550952" Nov 03 20:20:57.682041 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:20:57.682005 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq" Nov 03 20:20:58.276201 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:20:58.256934 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:21:00.552202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:00.546943 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:00.581202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:00.580642 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:00.742338 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:00.742274 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:00.949315 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:00.949271 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:00.999317 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:00.999267 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:01.129318 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:01.129270 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Nov 03 20:21:01.317311 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:01.287566 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" podUID="f7b8c27b-baaa-4eb1-89b7-263cc25d7a08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:01.558359 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:01.558289 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:01.559537 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:01.559509 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:21:01.661266 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:01.644979 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:01.663143 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:01.663115 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:21:01.992733 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:01.992637 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:21:01.993618 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:01.993590 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:21:02.049201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:02.029897 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:02.159226 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:02.099668 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9" podUID="8ad604d4-e127-4bd4-9d38-6f8b3987b313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:02.159226 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:02.149856 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:21:02.299386 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:02.299229 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:21:02.360909 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:02.354948 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" podUID="f7b8c27b-baaa-4eb1-89b7-263cc25d7a08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 03 20:21:02.360909 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:02.360296 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:21:02.465533 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:02.465503 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:21:02.503155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:02.503118 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:21:02.503443 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:02.503416 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:21:02.505140 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:02.505121 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:21:05.661325 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:05.661291 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25"] Nov 03 20:21:05.667846 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:05.667815 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25" Nov 03 20:21:05.673471 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:05.673438 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25"] Nov 03 20:21:05.746697 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:05.746659 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-7fd1a7-dockercfg-ml5rb\"" Nov 03 20:21:05.840103 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:05.747044 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-7fd1a7\"" Nov 03 20:21:05.840103 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:05.787493 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbstm\" (UniqueName: \"kubernetes.io/projected/803e64a2-6d50-49ac-aeef-7c7148e79e82-kube-api-access-zbstm\") pod \"payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25\" (UID: \"803e64a2-6d50-49ac-aeef-7c7148e79e82\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25" Nov 03 20:21:05.840103 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:05.835618 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803e64a2-6d50-49ac-aeef-7c7148e79e82-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25\" (UID: \"803e64a2-6d50-49ac-aeef-7c7148e79e82\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25" Nov 03 20:21:05.937282 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:05.937170 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbstm\" (UniqueName: \"kubernetes.io/projected/803e64a2-6d50-49ac-aeef-7c7148e79e82-kube-api-access-zbstm\") pod \"payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25\" (UID: \"803e64a2-6d50-49ac-aeef-7c7148e79e82\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25" Nov 03 20:21:05.939162 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:05.939135 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803e64a2-6d50-49ac-aeef-7c7148e79e82-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25\" (UID: \"803e64a2-6d50-49ac-aeef-7c7148e79e82\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25" Nov 03 20:21:05.940499 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:05.940160 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803e64a2-6d50-49ac-aeef-7c7148e79e82-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25\" (UID: \"803e64a2-6d50-49ac-aeef-7c7148e79e82\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25" Nov 03 20:21:06.007273 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:06.006310 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbstm\" (UniqueName: \"kubernetes.io/projected/803e64a2-6d50-49ac-aeef-7c7148e79e82-kube-api-access-zbstm\") pod \"payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25\" (UID: \"803e64a2-6d50-49ac-aeef-7c7148e79e82\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25" Nov 03 20:21:06.296937 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:06.296851 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25" Nov 03 20:21:06.596137 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:21:06.596109 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod803e64a2_6d50_49ac_aeef_7c7148e79e82.slice/crio-6f70a8bc13fd944cc89ae99827218c274b8f2061ef50a05912863c82b65cf8fa WatchSource:0}: Error finding container 6f70a8bc13fd944cc89ae99827218c274b8f2061ef50a05912863c82b65cf8fa: Status 404 returned error can't find the container with id 6f70a8bc13fd944cc89ae99827218c274b8f2061ef50a05912863c82b65cf8fa Nov 03 20:21:06.597447 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:06.597416 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25"] Nov 03 20:21:06.998795 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:06.998706 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc"] Nov 03 20:21:07.575262 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:07.575220 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25" event={"ID":"803e64a2-6d50-49ac-aeef-7c7148e79e82","Type":"ContainerStarted","Data":"b03988dfb85987b23dd3bbec0f85aa0a05acde8f0cf75ca70704405f365bb2a5"} Nov 03 20:21:07.575597 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:07.575567 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25" event={"ID":"803e64a2-6d50-49ac-aeef-7c7148e79e82","Type":"ContainerStarted","Data":"6f70a8bc13fd944cc89ae99827218c274b8f2061ef50a05912863c82b65cf8fa"} Nov 03 20:21:07.575723 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:07.575634 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25" Nov 03 20:21:07.610200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:07.597321 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25" podStartSLOduration=2.597300972 podStartE2EDuration="2.597300972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:07.59404056 +0000 UTC m=+1450.525908071" watchObservedRunningTime="2026-11-03 20:21:07.597300972 +0000 UTC m=+1450.529168484" Nov 03 20:21:08.227497 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:08.227460 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7081-2e149b"] Nov 03 20:21:08.249038 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:08.249006 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7081-2e149b"] Nov 03 20:21:08.249255 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:08.249122 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7081-2e149b" Nov 03 20:21:08.252970 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:08.252943 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dwjcb\"" Nov 03 20:21:08.321551 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:08.321515 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj9qj\" (UniqueName: \"kubernetes.io/projected/1b5d2c77-5c76-4d6d-82e7-c1bc67967557-kube-api-access-wj9qj\") pod \"test-curl-7081-2e149b\" (UID: \"1b5d2c77-5c76-4d6d-82e7-c1bc67967557\") " pod="opendatahub/test-curl-7081-2e149b" Nov 03 20:21:08.423165 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:08.423125 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj9qj\" (UniqueName: \"kubernetes.io/projected/1b5d2c77-5c76-4d6d-82e7-c1bc67967557-kube-api-access-wj9qj\") pod \"test-curl-7081-2e149b\" (UID: \"1b5d2c77-5c76-4d6d-82e7-c1bc67967557\") " pod="opendatahub/test-curl-7081-2e149b" Nov 03 20:21:08.435618 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:08.435592 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj9qj\" (UniqueName: \"kubernetes.io/projected/1b5d2c77-5c76-4d6d-82e7-c1bc67967557-kube-api-access-wj9qj\") pod \"test-curl-7081-2e149b\" (UID: \"1b5d2c77-5c76-4d6d-82e7-c1bc67967557\") " pod="opendatahub/test-curl-7081-2e149b" Nov 03 20:21:08.576473 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:08.576441 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7081-2e149b" Nov 03 20:21:08.896909 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:08.896881 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7081-2e149b"] Nov 03 20:21:08.897782 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:21:08.897751 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b5d2c77_5c76_4d6d_82e7_c1bc67967557.slice/crio-d8f3465df8aad16a298334532ed7816967ef0d0dbc1ce3a1bb8d99acfe8657b2 WatchSource:0}: Error finding container d8f3465df8aad16a298334532ed7816967ef0d0dbc1ce3a1bb8d99acfe8657b2: Status 404 returned error can't find the container with id d8f3465df8aad16a298334532ed7816967ef0d0dbc1ce3a1bb8d99acfe8657b2 Nov 03 20:21:09.595627 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:09.595583 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7081-2e149b" event={"ID":"1b5d2c77-5c76-4d6d-82e7-c1bc67967557","Type":"ContainerStarted","Data":"9b8fac917a1d55844ba48baeba2a4fd7938ba74549e627b499ba9f4dffad20c3"} Nov 03 20:21:09.595627 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:09.595638 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7081-2e149b" event={"ID":"1b5d2c77-5c76-4d6d-82e7-c1bc67967557","Type":"ContainerStarted","Data":"d8f3465df8aad16a298334532ed7816967ef0d0dbc1ce3a1bb8d99acfe8657b2"} Nov 03 20:21:09.613934 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:09.613885 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7081-2e149b" podStartSLOduration=1.61387133 podStartE2EDuration="1.61387133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:09.610897592 +0000 UTC m=+1452.542765103" watchObservedRunningTime="2026-11-03 20:21:09.61387133 +0000 UTC m=+1452.545738841" Nov 03 20:21:10.449612 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:10.448881 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7081-2e149b"] Nov 03 20:21:10.454836 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:10.454804 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7081-2e149b"] Nov 03 20:21:10.605479 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:10.605448 4918 generic.go:356] "Generic (PLEG): container finished" podID="1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" containerID="7320646a9c45f1fbc4bae0aed37c5df44d18947d80ea929fc253520e10799ba2" exitCode=0 Nov 03 20:21:10.605960 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:10.605525 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" event={"ID":"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7","Type":"ContainerDied","Data":"7320646a9c45f1fbc4bae0aed37c5df44d18947d80ea929fc253520e10799ba2"} Nov 03 20:21:10.606084 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:10.606066 4918 scope.go:122] "RemoveContainer" containerID="7320646a9c45f1fbc4bae0aed37c5df44d18947d80ea929fc253520e10799ba2" Nov 03 20:21:10.607748 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:10.607716 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:21:10.607821 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:10.607790 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:21:10.607856 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:10.607827 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:21:11.615536 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:11.615500 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" event={"ID":"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7","Type":"ContainerStarted","Data":"fdc3ac4d958ebed156e75a66d521365ebc3f4da98f8b258ce638b0378b7fbb17"} Nov 03 20:21:11.616006 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:11.615830 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7081-2e149b" podUID="1b5d2c77-5c76-4d6d-82e7-c1bc67967557" containerName="test-curl-7081-2e149b" containerID="cri-o://9b8fac917a1d55844ba48baeba2a4fd7938ba74549e627b499ba9f4dffad20c3" gracePeriod=2 Nov 03 20:21:11.999865 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:11.999745 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc" podUID="0c6c5c52-c436-40c7-86f8-0af529511e48" containerName="payload-processing" containerID="cri-o://3fb3195af16fadf99e5211bf72ae27eb2e28bd59c209e039e4d126a25bf36d13" gracePeriod=25 Nov 03 20:21:12.273584 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:12.273495 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:21:12.273763 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:12.273631 4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 20:21:12.273763 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:12.273675 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" podUID="1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 20:21:12.471369 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:12.471340 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc" Nov 03 20:21:12.514737 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:12.509307 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c6c5c52-c436-40c7-86f8-0af529511e48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c6c5c52-c436-40c7-86f8-0af529511e48-plugins-config-volume\") pod \"0c6c5c52-c436-40c7-86f8-0af529511e48\" (UID: \"0c6c5c52-c436-40c7-86f8-0af529511e48\") " Nov 03 20:21:12.514737 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:12.509405 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c6c5c52-c436-40c7-86f8-0af529511e48-kube-api-access-vppss\" (UniqueName: \"kubernetes.io/projected/0c6c5c52-c436-40c7-86f8-0af529511e48-kube-api-access-vppss\") pod \"0c6c5c52-c436-40c7-86f8-0af529511e48\" (UID: \"0c6c5c52-c436-40c7-86f8-0af529511e48\") " Nov 03 20:21:12.514737 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:12.511011 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6c5c52-c436-40c7-86f8-0af529511e48-plugins-config-volume" pod "0c6c5c52-c436-40c7-86f8-0af529511e48" (UID: "0c6c5c52-c436-40c7-86f8-0af529511e48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:12.514737 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:12.512084 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c6c5c52-c436-40c7-86f8-0af529511e48-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:12.558656 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:12.558581 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6c5c52-c436-40c7-86f8-0af529511e48-kube-api-access-vppss" pod "0c6c5c52-c436-40c7-86f8-0af529511e48" (UID: "0c6c5c52-c436-40c7-86f8-0af529511e48"). InnerVolumeSpecName "kube-api-access-vppss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:12.614255 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:12.614219 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vppss\" (UniqueName: \"kubernetes.io/projected/0c6c5c52-c436-40c7-86f8-0af529511e48-kube-api-access-vppss\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:12.652434 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:12.652399 4918 generic.go:356] "Generic (PLEG): container finished" podID="0c6c5c52-c436-40c7-86f8-0af529511e48" containerID="3fb3195af16fadf99e5211bf72ae27eb2e28bd59c209e039e4d126a25bf36d13" exitCode=0 Nov 03 20:21:12.652807 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:12.652472 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc" Nov 03 20:21:12.652807 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:12.652483 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc" event={"ID":"0c6c5c52-c436-40c7-86f8-0af529511e48","Type":"ContainerDied","Data":"3fb3195af16fadf99e5211bf72ae27eb2e28bd59c209e039e4d126a25bf36d13"} Nov 03 20:21:12.652807 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:12.652566 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc" event={"ID":"0c6c5c52-c436-40c7-86f8-0af529511e48","Type":"ContainerDied","Data":"d4554fe15f9e3b020e7b66f619a564ef1dc143d543512d2d50c935d66cd5f21f"} Nov 03 20:21:12.652807 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:12.652590 4918 scope.go:122] "RemoveContainer" containerID="3fb3195af16fadf99e5211bf72ae27eb2e28bd59c209e039e4d126a25bf36d13" Nov 03 20:21:12.664138 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:12.664122 4918 scope.go:122] "RemoveContainer" containerID="3fb3195af16fadf99e5211bf72ae27eb2e28bd59c209e039e4d126a25bf36d13" Nov 03 20:21:12.664432 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:21:12.664415 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb3195af16fadf99e5211bf72ae27eb2e28bd59c209e039e4d126a25bf36d13\": container with ID starting with 3fb3195af16fadf99e5211bf72ae27eb2e28bd59c209e039e4d126a25bf36d13 not found: ID does not exist" containerID="3fb3195af16fadf99e5211bf72ae27eb2e28bd59c209e039e4d126a25bf36d13" Nov 03 20:21:12.664499 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:12.664442 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb3195af16fadf99e5211bf72ae27eb2e28bd59c209e039e4d126a25bf36d13"} err="failed to get container status \"3fb3195af16fadf99e5211bf72ae27eb2e28bd59c209e039e4d126a25bf36d13\": rpc error: code = NotFound desc = could not find container \"3fb3195af16fadf99e5211bf72ae27eb2e28bd59c209e039e4d126a25bf36d13\": container with ID starting with 3fb3195af16fadf99e5211bf72ae27eb2e28bd59c209e039e4d126a25bf36d13 not found: ID does not exist" Nov 03 20:21:12.680630 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:12.680605 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc"] Nov 03 20:21:12.693168 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:12.693145 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9af0cd-fcddcb954-sd9zc"] Nov 03 20:21:13.217919 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:13.217886 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c6c5c52-c436-40c7-86f8-0af529511e48" path="/var/lib/kubelet/pods/0c6c5c52-c436-40c7-86f8-0af529511e48/volumes" Nov 03 20:21:13.273784 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:13.273747 4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 20:21:13.273973 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:13.273814 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" podUID="1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 20:21:13.905171 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:13.905145 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7081-2e149b" Nov 03 20:21:13.960753 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:13.960723 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b5d2c77-5c76-4d6d-82e7-c1bc67967557-kube-api-access-wj9qj\" (UniqueName: \"kubernetes.io/projected/1b5d2c77-5c76-4d6d-82e7-c1bc67967557-kube-api-access-wj9qj\") pod \"1b5d2c77-5c76-4d6d-82e7-c1bc67967557\" (UID: \"1b5d2c77-5c76-4d6d-82e7-c1bc67967557\") " Nov 03 20:21:13.962916 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:13.962884 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b5d2c77-5c76-4d6d-82e7-c1bc67967557-kube-api-access-wj9qj" pod "1b5d2c77-5c76-4d6d-82e7-c1bc67967557" (UID: "1b5d2c77-5c76-4d6d-82e7-c1bc67967557"). InnerVolumeSpecName "kube-api-access-wj9qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:14.062703 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:14.062669 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj9qj\" (UniqueName: \"kubernetes.io/projected/1b5d2c77-5c76-4d6d-82e7-c1bc67967557-kube-api-access-wj9qj\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:14.274142 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:14.274060 4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 20:21:14.274142 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:14.274118 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" podUID="1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 20:21:14.671217 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:14.671169 4918 generic.go:356] "Generic (PLEG): container finished" podID="1b5d2c77-5c76-4d6d-82e7-c1bc67967557" containerID="9b8fac917a1d55844ba48baeba2a4fd7938ba74549e627b499ba9f4dffad20c3" exitCode=137 Nov 03 20:21:14.671428 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:14.671234 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7081-2e149b" Nov 03 20:21:14.671428 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:14.671264 4918 scope.go:122] "RemoveContainer" containerID="9b8fac917a1d55844ba48baeba2a4fd7938ba74549e627b499ba9f4dffad20c3" Nov 03 20:21:14.682756 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:14.682734 4918 scope.go:122] "RemoveContainer" containerID="9b8fac917a1d55844ba48baeba2a4fd7938ba74549e627b499ba9f4dffad20c3" Nov 03 20:21:14.683039 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:21:14.683022 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b8fac917a1d55844ba48baeba2a4fd7938ba74549e627b499ba9f4dffad20c3\": container with ID starting with 9b8fac917a1d55844ba48baeba2a4fd7938ba74549e627b499ba9f4dffad20c3 not found: ID does not exist" containerID="9b8fac917a1d55844ba48baeba2a4fd7938ba74549e627b499ba9f4dffad20c3" Nov 03 20:21:14.683087 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:14.683046 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b8fac917a1d55844ba48baeba2a4fd7938ba74549e627b499ba9f4dffad20c3"} err="failed to get container status \"9b8fac917a1d55844ba48baeba2a4fd7938ba74549e627b499ba9f4dffad20c3\": rpc error: code = NotFound desc = could not find container \"9b8fac917a1d55844ba48baeba2a4fd7938ba74549e627b499ba9f4dffad20c3\": container with ID starting with 9b8fac917a1d55844ba48baeba2a4fd7938ba74549e627b499ba9f4dffad20c3 not found: ID does not exist" Nov 03 20:21:14.683636 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:21:14.683608 4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7081-2e149b\" is forbidden: User \"system:node:ip-10-0-139-53.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.us-east-2.compute.internal' and this object" podUID="1b5d2c77-5c76-4d6d-82e7-c1bc67967557" pod="opendatahub/test-curl-7081-2e149b" Nov 03 20:21:15.252797 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:15.252762 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b5d2c77-5c76-4d6d-82e7-c1bc67967557" path="/var/lib/kubelet/pods/1b5d2c77-5c76-4d6d-82e7-c1bc67967557/volumes" Nov 03 20:21:15.273253 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:15.273217 4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 20:21:15.273423 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:15.273276 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" podUID="1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 20:21:16.274017 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:16.273982 4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 20:21:16.274446 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:16.274036 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" podUID="1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 20:21:17.283157 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:17.283118 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:21:17.283680 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:17.283598 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:21:17.285166 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:17.285140 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:21:18.588194 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:18.588158 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25" Nov 03 20:21:28.221443 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:21:28.221409 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:21:47.813649 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:47.813564 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp"] Nov 03 20:21:47.814238 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:47.813787 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" podUID="0563a3f2-35d7-4d0a-a99d-b4d1e9277855" containerName="istio-proxy" containerID="cri-o://7bbd1b1dcbce6c44e04b2b4f72e842a40bd47ce1d3ef8c330737d3ab5b2dfc74" gracePeriod=30 Nov 03 20:21:47.819074 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:47.819045 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc"] Nov 03 20:21:47.819306 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:47.819284 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" podUID="461d7a11-e952-4fbf-a510-e1a83d824555" containerName="istio-proxy" containerID="cri-o://74df9c2e396000e1ae488117f0dce925935824ebfc85537b752c7f7738f3d7e0" gracePeriod=30 Nov 03 20:21:47.824060 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:47.824032 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48"] Nov 03 20:21:47.824267 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:47.824242 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" podUID="0553330c-c499-46c7-a5c4-0d4465e2d817" containerName="istio-proxy" containerID="cri-o://e2a4aff77761a7204bf48598e9d52a987035b53613f074cd4e46bdeb7f23bb44" gracePeriod=30 Nov 03 20:21:53.156167 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.156130 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:21:53.178684 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.178662 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:21:53.278330 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.278300 4918 generic.go:356] "Generic (PLEG): container finished" podID="461d7a11-e952-4fbf-a510-e1a83d824555" containerID="74df9c2e396000e1ae488117f0dce925935824ebfc85537b752c7f7738f3d7e0" exitCode=0 Nov 03 20:21:53.278464 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.278373 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" event={"ID":"461d7a11-e952-4fbf-a510-e1a83d824555","Type":"ContainerDied","Data":"74df9c2e396000e1ae488117f0dce925935824ebfc85537b752c7f7738f3d7e0"} Nov 03 20:21:53.279868 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.279841 4918 generic.go:356] "Generic (PLEG): container finished" podID="0563a3f2-35d7-4d0a-a99d-b4d1e9277855" containerID="7bbd1b1dcbce6c44e04b2b4f72e842a40bd47ce1d3ef8c330737d3ab5b2dfc74" exitCode=0 Nov 03 20:21:53.280021 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.279902 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" Nov 03 20:21:53.280021 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.279918 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" event={"ID":"0563a3f2-35d7-4d0a-a99d-b4d1e9277855","Type":"ContainerDied","Data":"7bbd1b1dcbce6c44e04b2b4f72e842a40bd47ce1d3ef8c330737d3ab5b2dfc74"} Nov 03 20:21:53.280021 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.279963 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp" event={"ID":"0563a3f2-35d7-4d0a-a99d-b4d1e9277855","Type":"ContainerDied","Data":"facd2aa75b70247e8a773b85b70f5e1273253e9170c8fefcf1bd3358af89824a"} Nov 03 20:21:53.280021 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.279989 4918 scope.go:122] "RemoveContainer" containerID="7bbd1b1dcbce6c44e04b2b4f72e842a40bd47ce1d3ef8c330737d3ab5b2dfc74" Nov 03 20:21:53.281378 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.281359 4918 generic.go:356] "Generic (PLEG): container finished" podID="0553330c-c499-46c7-a5c4-0d4465e2d817" containerID="e2a4aff77761a7204bf48598e9d52a987035b53613f074cd4e46bdeb7f23bb44" exitCode=0 Nov 03 20:21:53.281476 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.281398 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" Nov 03 20:21:53.281513 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.281400 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" event={"ID":"0553330c-c499-46c7-a5c4-0d4465e2d817","Type":"ContainerDied","Data":"e2a4aff77761a7204bf48598e9d52a987035b53613f074cd4e46bdeb7f23bb44"} Nov 03 20:21:53.281557 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.281526 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48" event={"ID":"0553330c-c499-46c7-a5c4-0d4465e2d817","Type":"ContainerDied","Data":"da3a8a738e1edf2688ec993c5d0de6931c6c2438f49985da7051087988337c63"} Nov 03 20:21:53.283751 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.283733 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-data\") pod \"0553330c-c499-46c7-a5c4-0d4465e2d817\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " Nov 03 20:21:53.283816 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.283762 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-workload-socket\") pod \"0553330c-c499-46c7-a5c4-0d4465e2d817\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " Nov 03 20:21:53.283864 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.283821 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-envoy\") pod \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " Nov 03 20:21:53.283864 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.283852 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-token\" (UniqueName: \"kubernetes.io/projected/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-token\") pod \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " Nov 03 20:21:53.283973 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.283919 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-data\") pod \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " Nov 03 20:21:53.283973 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.283958 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-kube-api-access-qngml\" (UniqueName: \"kubernetes.io/projected/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-kube-api-access-qngml\") pod \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " Nov 03 20:21:53.284078 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.284027 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-workload-socket\") pod \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " Nov 03 20:21:53.284078 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.284018 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-data" pod "0553330c-c499-46c7-a5c4-0d4465e2d817" (UID: "0553330c-c499-46c7-a5c4-0d4465e2d817"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:53.284201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.284081 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-podinfo\") pod \"0553330c-c499-46c7-a5c4-0d4465e2d817\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " Nov 03 20:21:53.284201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.284100 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-workload-socket" pod "0553330c-c499-46c7-a5c4-0d4465e2d817" (UID: "0553330c-c499-46c7-a5c4-0d4465e2d817"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:53.284201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.284147 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-token\" (UniqueName: \"kubernetes.io/projected/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-token\") pod \"0553330c-c499-46c7-a5c4-0d4465e2d817\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " Nov 03 20:21:53.284354 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.284203 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0553330c-c499-46c7-a5c4-0d4465e2d817-kube-api-access-7wqwb\" (UniqueName: \"kubernetes.io/projected/0553330c-c499-46c7-a5c4-0d4465e2d817-kube-api-access-7wqwb\") pod \"0553330c-c499-46c7-a5c4-0d4465e2d817\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " Nov 03 20:21:53.284354 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.284252 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-envoy\") pod \"0553330c-c499-46c7-a5c4-0d4465e2d817\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " Nov 03 20:21:53.284354 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.284306 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-data" pod "0563a3f2-35d7-4d0a-a99d-b4d1e9277855" (UID: "0563a3f2-35d7-4d0a-a99d-b4d1e9277855"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:53.284354 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.284344 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-workload-certs\") pod \"0553330c-c499-46c7-a5c4-0d4465e2d817\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " Nov 03 20:21:53.284578 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.284389 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-workload-certs\") pod \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " Nov 03 20:21:53.284578 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.284474 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istiod-ca-cert\") pod \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " Nov 03 20:21:53.284578 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.284522 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0553330c-c499-46c7-a5c4-0d4465e2d817-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0553330c-c499-46c7-a5c4-0d4465e2d817-istiod-ca-cert\") pod \"0553330c-c499-46c7-a5c4-0d4465e2d817\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " Nov 03 20:21:53.284578 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.284565 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-podinfo\") pod \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " Nov 03 20:21:53.284862 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.284635 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-credential-socket\") pod \"0553330c-c499-46c7-a5c4-0d4465e2d817\" (UID: \"0553330c-c499-46c7-a5c4-0d4465e2d817\") " Nov 03 20:21:53.284862 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.284677 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-credential-socket\") pod \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\" (UID: \"0563a3f2-35d7-4d0a-a99d-b4d1e9277855\") " Nov 03 20:21:53.284862 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.284752 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-workload-certs" pod "0553330c-c499-46c7-a5c4-0d4465e2d817" (UID: "0553330c-c499-46c7-a5c4-0d4465e2d817"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:53.284862 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.284775 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-workload-socket" pod "0563a3f2-35d7-4d0a-a99d-b4d1e9277855" (UID: "0563a3f2-35d7-4d0a-a99d-b4d1e9277855"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:53.285698 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.285672 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.285810 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.285702 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.285810 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.285717 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.285810 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.285734 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.285810 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.285749 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.286068 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.286020 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-credential-socket" pod "0563a3f2-35d7-4d0a-a99d-b4d1e9277855" (UID: "0563a3f2-35d7-4d0a-a99d-b4d1e9277855"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:53.286303 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.286277 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-workload-certs" pod "0563a3f2-35d7-4d0a-a99d-b4d1e9277855" (UID: "0563a3f2-35d7-4d0a-a99d-b4d1e9277855"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:53.289228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.286577 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-envoy" pod "0563a3f2-35d7-4d0a-a99d-b4d1e9277855" (UID: "0563a3f2-35d7-4d0a-a99d-b4d1e9277855"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:53.289228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.286582 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istiod-ca-cert" pod "0563a3f2-35d7-4d0a-a99d-b4d1e9277855" (UID: "0563a3f2-35d7-4d0a-a99d-b4d1e9277855"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:53.289228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.286639 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-token" pod "0553330c-c499-46c7-a5c4-0d4465e2d817" (UID: "0553330c-c499-46c7-a5c4-0d4465e2d817"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:53.289228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.286840 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-kube-api-access-qngml" pod "0563a3f2-35d7-4d0a-a99d-b4d1e9277855" (UID: "0563a3f2-35d7-4d0a-a99d-b4d1e9277855"). InnerVolumeSpecName "kube-api-access-qngml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:53.289228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.286841 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-credential-socket" pod "0553330c-c499-46c7-a5c4-0d4465e2d817" (UID: "0553330c-c499-46c7-a5c4-0d4465e2d817"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:53.289228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.286964 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0553330c-c499-46c7-a5c4-0d4465e2d817-istiod-ca-cert" pod "0553330c-c499-46c7-a5c4-0d4465e2d817" (UID: "0553330c-c499-46c7-a5c4-0d4465e2d817"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:53.289228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.287388 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-token" pod "0563a3f2-35d7-4d0a-a99d-b4d1e9277855" (UID: "0563a3f2-35d7-4d0a-a99d-b4d1e9277855"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:53.289228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.287880 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0553330c-c499-46c7-a5c4-0d4465e2d817-kube-api-access-7wqwb" pod "0553330c-c499-46c7-a5c4-0d4465e2d817" (UID: "0553330c-c499-46c7-a5c4-0d4465e2d817"). InnerVolumeSpecName "kube-api-access-7wqwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:53.289228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.287948 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-envoy" pod "0553330c-c499-46c7-a5c4-0d4465e2d817" (UID: "0553330c-c499-46c7-a5c4-0d4465e2d817"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:53.289228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.288790 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-podinfo" pod "0553330c-c499-46c7-a5c4-0d4465e2d817" (UID: "0553330c-c499-46c7-a5c4-0d4465e2d817"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:21:53.289788 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.289735 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-podinfo" pod "0563a3f2-35d7-4d0a-a99d-b4d1e9277855" (UID: "0563a3f2-35d7-4d0a-a99d-b4d1e9277855"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:21:53.293394 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.293377 4918 scope.go:122] "RemoveContainer" containerID="7bbd1b1dcbce6c44e04b2b4f72e842a40bd47ce1d3ef8c330737d3ab5b2dfc74" Nov 03 20:21:53.293681 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:21:53.293661 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bbd1b1dcbce6c44e04b2b4f72e842a40bd47ce1d3ef8c330737d3ab5b2dfc74\": container with ID starting with 7bbd1b1dcbce6c44e04b2b4f72e842a40bd47ce1d3ef8c330737d3ab5b2dfc74 not found: ID does not exist" containerID="7bbd1b1dcbce6c44e04b2b4f72e842a40bd47ce1d3ef8c330737d3ab5b2dfc74" Nov 03 20:21:53.293733 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.293691 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bbd1b1dcbce6c44e04b2b4f72e842a40bd47ce1d3ef8c330737d3ab5b2dfc74"} err="failed to get container status \"7bbd1b1dcbce6c44e04b2b4f72e842a40bd47ce1d3ef8c330737d3ab5b2dfc74\": rpc error: code = NotFound desc = could not find container \"7bbd1b1dcbce6c44e04b2b4f72e842a40bd47ce1d3ef8c330737d3ab5b2dfc74\": container with ID starting with 7bbd1b1dcbce6c44e04b2b4f72e842a40bd47ce1d3ef8c330737d3ab5b2dfc74 not found: ID does not exist" Nov 03 20:21:53.293733 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.293707 4918 scope.go:122] "RemoveContainer" containerID="e2a4aff77761a7204bf48598e9d52a987035b53613f074cd4e46bdeb7f23bb44" Nov 03 20:21:53.303038 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.303019 4918 scope.go:122] "RemoveContainer" containerID="e2a4aff77761a7204bf48598e9d52a987035b53613f074cd4e46bdeb7f23bb44" Nov 03 20:21:53.303362 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:21:53.303339 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2a4aff77761a7204bf48598e9d52a987035b53613f074cd4e46bdeb7f23bb44\": container with ID starting with e2a4aff77761a7204bf48598e9d52a987035b53613f074cd4e46bdeb7f23bb44 not found: ID does not exist" containerID="e2a4aff77761a7204bf48598e9d52a987035b53613f074cd4e46bdeb7f23bb44" Nov 03 20:21:53.303429 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.303370 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2a4aff77761a7204bf48598e9d52a987035b53613f074cd4e46bdeb7f23bb44"} err="failed to get container status \"e2a4aff77761a7204bf48598e9d52a987035b53613f074cd4e46bdeb7f23bb44\": rpc error: code = NotFound desc = could not find container \"e2a4aff77761a7204bf48598e9d52a987035b53613f074cd4e46bdeb7f23bb44\": container with ID starting with e2a4aff77761a7204bf48598e9d52a987035b53613f074cd4e46bdeb7f23bb44 not found: ID does not exist" Nov 03 20:21:53.387257 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.387226 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.387257 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.387252 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.387257 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.387263 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qngml\" (UniqueName: \"kubernetes.io/projected/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-kube-api-access-qngml\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.387257 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.387273 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.387257 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.387283 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.387257 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.387294 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wqwb\" (UniqueName: \"kubernetes.io/projected/0553330c-c499-46c7-a5c4-0d4465e2d817-kube-api-access-7wqwb\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.387257 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.387303 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.387257 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.387311 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.387257 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.387320 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.387257 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.387328 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0553330c-c499-46c7-a5c4-0d4465e2d817-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.387924 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.387336 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.387924 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.387345 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0553330c-c499-46c7-a5c4-0d4465e2d817-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.387924 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.387355 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0563a3f2-35d7-4d0a-a99d-b4d1e9277855-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.528855 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.528828 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:21:53.614308 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.614274 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48"] Nov 03 20:21:53.627880 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.627829 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-tlt48"] Nov 03 20:21:53.637865 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.637762 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp"] Nov 03 20:21:53.641839 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.641814 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-krrbp"] Nov 03 20:21:53.691499 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.691450 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/461d7a11-e952-4fbf-a510-e1a83d824555-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/461d7a11-e952-4fbf-a510-e1a83d824555-istio-podinfo\") pod \"461d7a11-e952-4fbf-a510-e1a83d824555\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " Nov 03 20:21:53.691971 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.691940 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-credential-socket\") pod \"461d7a11-e952-4fbf-a510-e1a83d824555\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " Nov 03 20:21:53.692147 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.692132 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-istio-data\") pod \"461d7a11-e952-4fbf-a510-e1a83d824555\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " Nov 03 20:21:53.692394 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.692373 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-istio-envoy\") pod \"461d7a11-e952-4fbf-a510-e1a83d824555\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " Nov 03 20:21:53.699921 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.699877 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/461d7a11-e952-4fbf-a510-e1a83d824555-istio-token\" (UniqueName: \"kubernetes.io/projected/461d7a11-e952-4fbf-a510-e1a83d824555-istio-token\") pod \"461d7a11-e952-4fbf-a510-e1a83d824555\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " Nov 03 20:21:53.700097 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.699960 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-workload-socket\") pod \"461d7a11-e952-4fbf-a510-e1a83d824555\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " Nov 03 20:21:53.700097 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.700015 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-workload-certs\") pod \"461d7a11-e952-4fbf-a510-e1a83d824555\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " Nov 03 20:21:53.700097 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.700085 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/461d7a11-e952-4fbf-a510-e1a83d824555-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/461d7a11-e952-4fbf-a510-e1a83d824555-istiod-ca-cert\") pod \"461d7a11-e952-4fbf-a510-e1a83d824555\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " Nov 03 20:21:53.700394 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.700226 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/461d7a11-e952-4fbf-a510-e1a83d824555-kube-api-access-vz6j6\" (UniqueName: \"kubernetes.io/projected/461d7a11-e952-4fbf-a510-e1a83d824555-kube-api-access-vz6j6\") pod \"461d7a11-e952-4fbf-a510-e1a83d824555\" (UID: \"461d7a11-e952-4fbf-a510-e1a83d824555\") " Nov 03 20:21:53.703124 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.702670 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-istio-data" pod "461d7a11-e952-4fbf-a510-e1a83d824555" (UID: "461d7a11-e952-4fbf-a510-e1a83d824555"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:53.703124 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.702884 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-workload-certs" pod "461d7a11-e952-4fbf-a510-e1a83d824555" (UID: "461d7a11-e952-4fbf-a510-e1a83d824555"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:53.703124 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.703044 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-workload-socket" pod "461d7a11-e952-4fbf-a510-e1a83d824555" (UID: "461d7a11-e952-4fbf-a510-e1a83d824555"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:53.709064 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.705579 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-credential-socket" pod "461d7a11-e952-4fbf-a510-e1a83d824555" (UID: "461d7a11-e952-4fbf-a510-e1a83d824555"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:53.709064 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.706112 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/461d7a11-e952-4fbf-a510-e1a83d824555-istiod-ca-cert" pod "461d7a11-e952-4fbf-a510-e1a83d824555" (UID: "461d7a11-e952-4fbf-a510-e1a83d824555"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:53.709064 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.706416 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.709064 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.706443 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.709064 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.706457 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/461d7a11-e952-4fbf-a510-e1a83d824555-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.709064 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.706471 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.709064 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.706484 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.710170 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.710134 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461d7a11-e952-4fbf-a510-e1a83d824555-kube-api-access-vz6j6" pod "461d7a11-e952-4fbf-a510-e1a83d824555" (UID: "461d7a11-e952-4fbf-a510-e1a83d824555"). InnerVolumeSpecName "kube-api-access-vz6j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:53.711626 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.711576 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/461d7a11-e952-4fbf-a510-e1a83d824555-istio-podinfo" pod "461d7a11-e952-4fbf-a510-e1a83d824555" (UID: "461d7a11-e952-4fbf-a510-e1a83d824555"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:21:53.711828 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.711770 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461d7a11-e952-4fbf-a510-e1a83d824555-istio-token" pod "461d7a11-e952-4fbf-a510-e1a83d824555" (UID: "461d7a11-e952-4fbf-a510-e1a83d824555"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:53.713383 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.713364 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-istio-envoy" pod "461d7a11-e952-4fbf-a510-e1a83d824555" (UID: "461d7a11-e952-4fbf-a510-e1a83d824555"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:53.808205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.808158 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz6j6\" (UniqueName: \"kubernetes.io/projected/461d7a11-e952-4fbf-a510-e1a83d824555-kube-api-access-vz6j6\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.808205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.808207 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/461d7a11-e952-4fbf-a510-e1a83d824555-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.808668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.808223 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/461d7a11-e952-4fbf-a510-e1a83d824555-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.808668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.808232 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/461d7a11-e952-4fbf-a510-e1a83d824555-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.875828 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:53.875759 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d"] Nov 03 20:21:54.379649 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:54.379615 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" Nov 03 20:21:54.380599 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:54.380450 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc" event={"ID":"461d7a11-e952-4fbf-a510-e1a83d824555","Type":"ContainerDied","Data":"3dee98bbc9ab322969bc4b2e2b57c3ca3be50710088f875c28f18c83405a8de4"} Nov 03 20:21:54.380722 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:54.380651 4918 scope.go:122] "RemoveContainer" containerID="74df9c2e396000e1ae488117f0dce925935824ebfc85537b752c7f7738f3d7e0" Nov 03 20:21:54.413699 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:54.413650 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc"] Nov 03 20:21:54.418930 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:54.418902 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-w8ccc"] Nov 03 20:21:55.220590 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:55.220556 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0553330c-c499-46c7-a5c4-0d4465e2d817" path="/var/lib/kubelet/pods/0553330c-c499-46c7-a5c4-0d4465e2d817/volumes" Nov 03 20:21:55.221155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:55.221132 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0563a3f2-35d7-4d0a-a99d-b4d1e9277855" path="/var/lib/kubelet/pods/0563a3f2-35d7-4d0a-a99d-b4d1e9277855/volumes" Nov 03 20:21:55.221780 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:55.221750 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="461d7a11-e952-4fbf-a510-e1a83d824555" path="/var/lib/kubelet/pods/461d7a11-e952-4fbf-a510-e1a83d824555/volumes" Nov 03 20:21:57.415092 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:57.415058 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:21:57.418669 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:57.418642 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:21:58.223588 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:21:58.223551 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:21:58.730367 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:58.730330 4918 scope.go:122] "RemoveContainer" containerID="c85f2497a8d10944c107d56f964f9183781adbc1e2919fad93ff55420eb85f3c" Nov 03 20:21:58.892202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:58.891509 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d" podUID="9b6760f6-b0a4-4549-a135-e3643d29e2e6" containerName="payload-pre-processing" containerID="cri-o://a063aedc2822e4134cd0768a8bfc9c2e2df19e6a3eae615799d94d49e181d2be" gracePeriod=25 Nov 03 20:21:59.208125 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:59.208097 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d" Nov 03 20:21:59.309759 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:59.309661 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b6760f6-b0a4-4549-a135-e3643d29e2e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b6760f6-b0a4-4549-a135-e3643d29e2e6-plugins-config-volume\") pod \"9b6760f6-b0a4-4549-a135-e3643d29e2e6\" (UID: \"9b6760f6-b0a4-4549-a135-e3643d29e2e6\") " Nov 03 20:21:59.309958 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:59.309874 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b6760f6-b0a4-4549-a135-e3643d29e2e6-kube-api-access-mpttk\" (UniqueName: \"kubernetes.io/projected/9b6760f6-b0a4-4549-a135-e3643d29e2e6-kube-api-access-mpttk\") pod \"9b6760f6-b0a4-4549-a135-e3643d29e2e6\" (UID: \"9b6760f6-b0a4-4549-a135-e3643d29e2e6\") " Nov 03 20:21:59.310642 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:59.310569 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b6760f6-b0a4-4549-a135-e3643d29e2e6-plugins-config-volume" pod "9b6760f6-b0a4-4549-a135-e3643d29e2e6" (UID: "9b6760f6-b0a4-4549-a135-e3643d29e2e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:59.311123 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:59.311084 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b6760f6-b0a4-4549-a135-e3643d29e2e6-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:59.326292 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:59.326260 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b6760f6-b0a4-4549-a135-e3643d29e2e6-kube-api-access-mpttk" pod "9b6760f6-b0a4-4549-a135-e3643d29e2e6" (UID: "9b6760f6-b0a4-4549-a135-e3643d29e2e6"). InnerVolumeSpecName "kube-api-access-mpttk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:59.413044 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:59.413005 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpttk\" (UniqueName: \"kubernetes.io/projected/9b6760f6-b0a4-4549-a135-e3643d29e2e6-kube-api-access-mpttk\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:59.458562 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:59.458528 4918 generic.go:356] "Generic (PLEG): container finished" podID="9b6760f6-b0a4-4549-a135-e3643d29e2e6" containerID="a063aedc2822e4134cd0768a8bfc9c2e2df19e6a3eae615799d94d49e181d2be" exitCode=0 Nov 03 20:21:59.458761 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:59.458580 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d" Nov 03 20:21:59.458761 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:59.458626 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d" event={"ID":"9b6760f6-b0a4-4549-a135-e3643d29e2e6","Type":"ContainerDied","Data":"a063aedc2822e4134cd0768a8bfc9c2e2df19e6a3eae615799d94d49e181d2be"} Nov 03 20:21:59.458761 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:59.458692 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d" event={"ID":"9b6760f6-b0a4-4549-a135-e3643d29e2e6","Type":"ContainerDied","Data":"111009de49e4f8554d953c5e2ea015a178db0ba638f6339f2bd4b957ad009939"} Nov 03 20:21:59.458761 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:59.458739 4918 scope.go:122] "RemoveContainer" containerID="a063aedc2822e4134cd0768a8bfc9c2e2df19e6a3eae615799d94d49e181d2be" Nov 03 20:21:59.469910 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:59.469892 4918 scope.go:122] "RemoveContainer" containerID="a063aedc2822e4134cd0768a8bfc9c2e2df19e6a3eae615799d94d49e181d2be" Nov 03 20:21:59.470201 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:21:59.470160 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a063aedc2822e4134cd0768a8bfc9c2e2df19e6a3eae615799d94d49e181d2be\": container with ID starting with a063aedc2822e4134cd0768a8bfc9c2e2df19e6a3eae615799d94d49e181d2be not found: ID does not exist" containerID="a063aedc2822e4134cd0768a8bfc9c2e2df19e6a3eae615799d94d49e181d2be" Nov 03 20:21:59.470306 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:59.470211 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a063aedc2822e4134cd0768a8bfc9c2e2df19e6a3eae615799d94d49e181d2be"} err="failed to get container status \"a063aedc2822e4134cd0768a8bfc9c2e2df19e6a3eae615799d94d49e181d2be\": rpc error: code = NotFound desc = could not find container \"a063aedc2822e4134cd0768a8bfc9c2e2df19e6a3eae615799d94d49e181d2be\": container with ID starting with a063aedc2822e4134cd0768a8bfc9c2e2df19e6a3eae615799d94d49e181d2be not found: ID does not exist" Nov 03 20:21:59.483638 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:59.483605 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d"] Nov 03 20:21:59.487868 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:21:59.487842 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06294c6a-54dc8cc9ff-2td5d"] Nov 03 20:22:01.217441 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:01.217405 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b6760f6-b0a4-4549-a135-e3643d29e2e6" path="/var/lib/kubelet/pods/9b6760f6-b0a4-4549-a135-e3643d29e2e6/volumes" Nov 03 20:22:11.840818 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:11.840780 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq"] Nov 03 20:22:16.841653 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:16.841607 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq" podUID="76ff8461-6882-4ad5-8d54-eb63951fb83c" containerName="payload-pre-processing" containerID="cri-o://6ac53108b269a56401145bd5743b5efb7c460cbbef31a030e08ef33cdae12034" gracePeriod=25 Nov 03 20:22:17.126266 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:17.126242 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq" Nov 03 20:22:17.224327 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:17.224297 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ff8461-6882-4ad5-8d54-eb63951fb83c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ff8461-6882-4ad5-8d54-eb63951fb83c-plugins-config-volume\") pod \"76ff8461-6882-4ad5-8d54-eb63951fb83c\" (UID: \"76ff8461-6882-4ad5-8d54-eb63951fb83c\") " Nov 03 20:22:17.224474 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:17.224424 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ff8461-6882-4ad5-8d54-eb63951fb83c-kube-api-access-7hhcw\" (UniqueName: \"kubernetes.io/projected/76ff8461-6882-4ad5-8d54-eb63951fb83c-kube-api-access-7hhcw\") pod \"76ff8461-6882-4ad5-8d54-eb63951fb83c\" (UID: \"76ff8461-6882-4ad5-8d54-eb63951fb83c\") " Nov 03 20:22:17.224654 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:17.224632 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ff8461-6882-4ad5-8d54-eb63951fb83c-plugins-config-volume" pod "76ff8461-6882-4ad5-8d54-eb63951fb83c" (UID: "76ff8461-6882-4ad5-8d54-eb63951fb83c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:17.224949 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:17.224934 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ff8461-6882-4ad5-8d54-eb63951fb83c-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:17.226636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:17.226611 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ff8461-6882-4ad5-8d54-eb63951fb83c-kube-api-access-7hhcw" pod "76ff8461-6882-4ad5-8d54-eb63951fb83c" (UID: "76ff8461-6882-4ad5-8d54-eb63951fb83c"). InnerVolumeSpecName "kube-api-access-7hhcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:17.326277 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:17.326246 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hhcw\" (UniqueName: \"kubernetes.io/projected/76ff8461-6882-4ad5-8d54-eb63951fb83c-kube-api-access-7hhcw\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:17.600824 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:17.600785 4918 generic.go:356] "Generic (PLEG): container finished" podID="76ff8461-6882-4ad5-8d54-eb63951fb83c" containerID="6ac53108b269a56401145bd5743b5efb7c460cbbef31a030e08ef33cdae12034" exitCode=0 Nov 03 20:22:17.601000 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:17.600832 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq" Nov 03 20:22:17.601000 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:17.600862 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq" event={"ID":"76ff8461-6882-4ad5-8d54-eb63951fb83c","Type":"ContainerDied","Data":"6ac53108b269a56401145bd5743b5efb7c460cbbef31a030e08ef33cdae12034"} Nov 03 20:22:17.601000 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:17.600909 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq" event={"ID":"76ff8461-6882-4ad5-8d54-eb63951fb83c","Type":"ContainerDied","Data":"add566e6228a7e6b1ac70f6cea4c858efce6496cbb6f8ea41ce8f251d091987c"} Nov 03 20:22:17.601000 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:17.600935 4918 scope.go:122] "RemoveContainer" containerID="6ac53108b269a56401145bd5743b5efb7c460cbbef31a030e08ef33cdae12034" Nov 03 20:22:17.611762 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:17.611744 4918 scope.go:122] "RemoveContainer" containerID="6ac53108b269a56401145bd5743b5efb7c460cbbef31a030e08ef33cdae12034" Nov 03 20:22:17.612037 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:22:17.612014 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ac53108b269a56401145bd5743b5efb7c460cbbef31a030e08ef33cdae12034\": container with ID starting with 6ac53108b269a56401145bd5743b5efb7c460cbbef31a030e08ef33cdae12034 not found: ID does not exist" containerID="6ac53108b269a56401145bd5743b5efb7c460cbbef31a030e08ef33cdae12034" Nov 03 20:22:17.612091 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:17.612045 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ac53108b269a56401145bd5743b5efb7c460cbbef31a030e08ef33cdae12034"} err="failed to get container status \"6ac53108b269a56401145bd5743b5efb7c460cbbef31a030e08ef33cdae12034\": rpc error: code = NotFound desc = could not find container \"6ac53108b269a56401145bd5743b5efb7c460cbbef31a030e08ef33cdae12034\": container with ID starting with 6ac53108b269a56401145bd5743b5efb7c460cbbef31a030e08ef33cdae12034 not found: ID does not exist" Nov 03 20:22:17.636645 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:17.636596 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq"] Nov 03 20:22:17.640973 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:17.640942 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d5927-7dd69b579f-plnwq"] Nov 03 20:22:19.217001 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:19.216963 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ff8461-6882-4ad5-8d54-eb63951fb83c" path="/var/lib/kubelet/pods/76ff8461-6882-4ad5-8d54-eb63951fb83c/volumes" Nov 03 20:22:28.209732 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:22:28.209698 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:22:38.233227 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.233190 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8"] Nov 03 20:22:38.234238 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234193 4918 cpu_manager.go:414] "containerMap: removing container" podUID="0c6c5c52-c436-40c7-86f8-0af529511e48" containerName="payload-processing" Nov 03 20:22:38.234238 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234239 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c6c5c52-c436-40c7-86f8-0af529511e48" containerName="payload-processing" Nov 03 20:22:38.234572 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234263 4918 cpu_manager.go:414] "containerMap: removing container" podUID="461d7a11-e952-4fbf-a510-e1a83d824555" containerName="istio-proxy" Nov 03 20:22:38.234572 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234271 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="461d7a11-e952-4fbf-a510-e1a83d824555" containerName="istio-proxy" Nov 03 20:22:38.234572 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234331 4918 cpu_manager.go:414] "containerMap: removing container" podUID="9b6760f6-b0a4-4549-a135-e3643d29e2e6" containerName="payload-pre-processing" Nov 03 20:22:38.234572 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234342 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b6760f6-b0a4-4549-a135-e3643d29e2e6" containerName="payload-pre-processing" Nov 03 20:22:38.234572 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234361 4918 cpu_manager.go:414] "containerMap: removing container" podUID="76ff8461-6882-4ad5-8d54-eb63951fb83c" containerName="payload-pre-processing" Nov 03 20:22:38.234572 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234368 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ff8461-6882-4ad5-8d54-eb63951fb83c" containerName="payload-pre-processing" Nov 03 20:22:38.234572 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234406 4918 cpu_manager.go:414] "containerMap: removing container" podUID="0553330c-c499-46c7-a5c4-0d4465e2d817" containerName="istio-proxy" Nov 03 20:22:38.234572 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234414 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0553330c-c499-46c7-a5c4-0d4465e2d817" containerName="istio-proxy" Nov 03 20:22:38.234572 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234439 4918 cpu_manager.go:414] "containerMap: removing container" podUID="0563a3f2-35d7-4d0a-a99d-b4d1e9277855" containerName="istio-proxy" Nov 03 20:22:38.234572 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234447 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0563a3f2-35d7-4d0a-a99d-b4d1e9277855" containerName="istio-proxy" Nov 03 20:22:38.234572 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234483 4918 cpu_manager.go:414] "containerMap: removing container" podUID="1b5d2c77-5c76-4d6d-82e7-c1bc67967557" containerName="test-curl-7081-2e149b" Nov 03 20:22:38.234572 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234491 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b5d2c77-5c76-4d6d-82e7-c1bc67967557" containerName="test-curl-7081-2e149b" Nov 03 20:22:38.235231 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234714 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b6760f6-b0a4-4549-a135-e3643d29e2e6" containerName="payload-pre-processing" Nov 03 20:22:38.235231 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234728 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ff8461-6882-4ad5-8d54-eb63951fb83c" containerName="payload-pre-processing" Nov 03 20:22:38.235231 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234737 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0553330c-c499-46c7-a5c4-0d4465e2d817" containerName="istio-proxy" Nov 03 20:22:38.235231 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234749 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0563a3f2-35d7-4d0a-a99d-b4d1e9277855" containerName="istio-proxy" Nov 03 20:22:38.235231 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234760 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b5d2c77-5c76-4d6d-82e7-c1bc67967557" containerName="test-curl-7081-2e149b" Nov 03 20:22:38.235231 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234772 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c6c5c52-c436-40c7-86f8-0af529511e48" containerName="payload-processing" Nov 03 20:22:38.235231 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.234779 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="461d7a11-e952-4fbf-a510-e1a83d824555" containerName="istio-proxy" Nov 03 20:22:38.238889 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.238868 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8" Nov 03 20:22:38.290448 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.290413 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8"] Nov 03 20:22:38.334235 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.334203 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5ca474-dockercfg-p4n2l\"" Nov 03 20:22:38.334580 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.334556 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5ca474\"" Nov 03 20:22:38.402201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.401745 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgtlm\" (UniqueName: \"kubernetes.io/projected/3b00acc2-340d-4fec-bab6-efadbf64e74d-kube-api-access-tgtlm\") pod \"payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8\" (UID: \"3b00acc2-340d-4fec-bab6-efadbf64e74d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8" Nov 03 20:22:38.402201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.402104 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b00acc2-340d-4fec-bab6-efadbf64e74d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8\" (UID: \"3b00acc2-340d-4fec-bab6-efadbf64e74d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8" Nov 03 20:22:38.510251 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.508729 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b00acc2-340d-4fec-bab6-efadbf64e74d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8\" (UID: \"3b00acc2-340d-4fec-bab6-efadbf64e74d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8" Nov 03 20:22:38.510251 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.508799 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgtlm\" (UniqueName: \"kubernetes.io/projected/3b00acc2-340d-4fec-bab6-efadbf64e74d-kube-api-access-tgtlm\") pod \"payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8\" (UID: \"3b00acc2-340d-4fec-bab6-efadbf64e74d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8" Nov 03 20:22:38.510251 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.510050 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b00acc2-340d-4fec-bab6-efadbf64e74d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8\" (UID: \"3b00acc2-340d-4fec-bab6-efadbf64e74d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8" Nov 03 20:22:38.579203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.575979 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgtlm\" (UniqueName: \"kubernetes.io/projected/3b00acc2-340d-4fec-bab6-efadbf64e74d-kube-api-access-tgtlm\") pod \"payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8\" (UID: \"3b00acc2-340d-4fec-bab6-efadbf64e74d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8" Nov 03 20:22:38.601203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.596319 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8" Nov 03 20:22:38.920445 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:38.920400 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8"] Nov 03 20:22:38.920785 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:22:38.920752 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b00acc2_340d_4fec_bab6_efadbf64e74d.slice/crio-3f0132287cf9b4d20bd505f2d10f95b0aa7ca91681fc109a89c56cd71ae70401 WatchSource:0}: Error finding container 3f0132287cf9b4d20bd505f2d10f95b0aa7ca91681fc109a89c56cd71ae70401: Status 404 returned error can't find the container with id 3f0132287cf9b4d20bd505f2d10f95b0aa7ca91681fc109a89c56cd71ae70401 Nov 03 20:22:39.823677 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:39.823640 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8" event={"ID":"3b00acc2-340d-4fec-bab6-efadbf64e74d","Type":"ContainerStarted","Data":"bc79b38baca616aaf2ed70277f71ab50d22358422abd1271a95cfef8f52e6551"} Nov 03 20:22:39.824161 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:39.823698 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8" event={"ID":"3b00acc2-340d-4fec-bab6-efadbf64e74d","Type":"ContainerStarted","Data":"3f0132287cf9b4d20bd505f2d10f95b0aa7ca91681fc109a89c56cd71ae70401"} Nov 03 20:22:39.824161 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:39.823721 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8" Nov 03 20:22:39.848733 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:39.848684 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8" podStartSLOduration=1.848670227 podStartE2EDuration="1.848670227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:39.84248281 +0000 UTC m=+1542.774350358" watchObservedRunningTime="2026-11-03 20:22:39.848670227 +0000 UTC m=+1542.780537738" Nov 03 20:22:42.432652 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.432619 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d"] Nov 03 20:22:42.437403 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.437382 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d" Nov 03 20:22:42.445670 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.445649 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5de824\"" Nov 03 20:22:42.455819 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.455793 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d"] Nov 03 20:22:42.471264 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.471240 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2"] Nov 03 20:22:42.476447 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.476425 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2" Nov 03 20:22:42.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.485615 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5de824-dockercfg-6vbrh\"" Nov 03 20:22:42.490852 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.490825 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2"] Nov 03 20:22:42.496132 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.496106 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7g25\" (UniqueName: \"kubernetes.io/projected/26102cd8-e567-4e90-acf5-a57fc2b0dae4-kube-api-access-r7g25\") pod \"payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d\" (UID: \"26102cd8-e567-4e90-acf5-a57fc2b0dae4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d" Nov 03 20:22:42.496246 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.496219 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26102cd8-e567-4e90-acf5-a57fc2b0dae4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d\" (UID: \"26102cd8-e567-4e90-acf5-a57fc2b0dae4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d" Nov 03 20:22:42.639198 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.630459 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7g25\" (UniqueName: \"kubernetes.io/projected/26102cd8-e567-4e90-acf5-a57fc2b0dae4-kube-api-access-r7g25\") pod \"payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d\" (UID: \"26102cd8-e567-4e90-acf5-a57fc2b0dae4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d" Nov 03 20:22:42.639198 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.630539 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz8xp\" (UniqueName: \"kubernetes.io/projected/3f3259f2-a529-4ea7-a637-b39c29e0a72f-kube-api-access-bz8xp\") pod \"payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2\" (UID: \"3f3259f2-a529-4ea7-a637-b39c29e0a72f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2" Nov 03 20:22:42.639198 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.630634 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3259f2-a529-4ea7-a637-b39c29e0a72f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2\" (UID: \"3f3259f2-a529-4ea7-a637-b39c29e0a72f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2" Nov 03 20:22:42.639198 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.630682 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26102cd8-e567-4e90-acf5-a57fc2b0dae4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d\" (UID: \"26102cd8-e567-4e90-acf5-a57fc2b0dae4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d" Nov 03 20:22:42.639198 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.637118 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26102cd8-e567-4e90-acf5-a57fc2b0dae4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d\" (UID: \"26102cd8-e567-4e90-acf5-a57fc2b0dae4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d" Nov 03 20:22:42.671265 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.653762 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7g25\" (UniqueName: \"kubernetes.io/projected/26102cd8-e567-4e90-acf5-a57fc2b0dae4-kube-api-access-r7g25\") pod \"payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d\" (UID: \"26102cd8-e567-4e90-acf5-a57fc2b0dae4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d" Nov 03 20:22:42.735020 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.734822 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz8xp\" (UniqueName: \"kubernetes.io/projected/3f3259f2-a529-4ea7-a637-b39c29e0a72f-kube-api-access-bz8xp\") pod \"payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2\" (UID: \"3f3259f2-a529-4ea7-a637-b39c29e0a72f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2" Nov 03 20:22:42.735020 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.734932 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3259f2-a529-4ea7-a637-b39c29e0a72f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2\" (UID: \"3f3259f2-a529-4ea7-a637-b39c29e0a72f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2" Nov 03 20:22:42.753904 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.753049 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d" Nov 03 20:22:42.754297 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.754234 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3259f2-a529-4ea7-a637-b39c29e0a72f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2\" (UID: \"3f3259f2-a529-4ea7-a637-b39c29e0a72f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2" Nov 03 20:22:42.791747 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.787775 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz8xp\" (UniqueName: \"kubernetes.io/projected/3f3259f2-a529-4ea7-a637-b39c29e0a72f-kube-api-access-bz8xp\") pod \"payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2\" (UID: \"3f3259f2-a529-4ea7-a637-b39c29e0a72f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2" Nov 03 20:22:42.809202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:42.808469 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2" Nov 03 20:22:43.166579 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:22:43.166550 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26102cd8_e567_4e90_acf5_a57fc2b0dae4.slice/crio-1c97d78ca394db084b7b987372ea4c5cfe8cf4f405c996bca171926d3786184f WatchSource:0}: Error finding container 1c97d78ca394db084b7b987372ea4c5cfe8cf4f405c996bca171926d3786184f: Status 404 returned error can't find the container with id 1c97d78ca394db084b7b987372ea4c5cfe8cf4f405c996bca171926d3786184f Nov 03 20:22:43.167270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:43.167104 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d"] Nov 03 20:22:43.169092 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:22:43.169070 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3259f2_a529_4ea7_a637_b39c29e0a72f.slice/crio-4d30d9fdb0012f41bde9f6a9f6861551357dbe69c5cb71efde89d158108a92b7 WatchSource:0}: Error finding container 4d30d9fdb0012f41bde9f6a9f6861551357dbe69c5cb71efde89d158108a92b7: Status 404 returned error can't find the container with id 4d30d9fdb0012f41bde9f6a9f6861551357dbe69c5cb71efde89d158108a92b7 Nov 03 20:22:43.172880 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:43.172854 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2"] Nov 03 20:22:43.878255 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:43.878211 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d" event={"ID":"26102cd8-e567-4e90-acf5-a57fc2b0dae4","Type":"ContainerStarted","Data":"fd355472123a5b5cb277200bc340c149b1ae776ae0ddf6dd2932f5f3b3720e5e"} Nov 03 20:22:43.878762 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:43.878316 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d" Nov 03 20:22:43.878762 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:43.878349 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d" event={"ID":"26102cd8-e567-4e90-acf5-a57fc2b0dae4","Type":"ContainerStarted","Data":"1c97d78ca394db084b7b987372ea4c5cfe8cf4f405c996bca171926d3786184f"} Nov 03 20:22:43.880090 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:43.880061 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2" event={"ID":"3f3259f2-a529-4ea7-a637-b39c29e0a72f","Type":"ContainerStarted","Data":"734e2b83b1aba2a48a01ef2271cbae38cb988944dadd60d6ebfbe2164e81bc75"} Nov 03 20:22:43.880226 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:43.880112 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2" event={"ID":"3f3259f2-a529-4ea7-a637-b39c29e0a72f","Type":"ContainerStarted","Data":"4d30d9fdb0012f41bde9f6a9f6861551357dbe69c5cb71efde89d158108a92b7"} Nov 03 20:22:43.880226 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:43.880150 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2" Nov 03 20:22:43.901684 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:43.901636 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d" podStartSLOduration=1.9016242669999999 podStartE2EDuration="1.901624267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:43.89580324 +0000 UTC m=+1546.827670780" watchObservedRunningTime="2026-11-03 20:22:43.901624267 +0000 UTC m=+1546.833491777" Nov 03 20:22:43.914783 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:43.914685 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2" podStartSLOduration=1.914669397 podStartE2EDuration="1.914669397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:43.913732481 +0000 UTC m=+1546.845599992" watchObservedRunningTime="2026-11-03 20:22:43.914669397 +0000 UTC m=+1546.846536909" Nov 03 20:22:49.315944 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:49.315905 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7081-00f201"] Nov 03 20:22:49.320699 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:49.320669 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7081-00f201" Nov 03 20:22:49.329133 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:49.329099 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7081-00f201"] Nov 03 20:22:49.330478 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:49.330453 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dwjcb\"" Nov 03 20:22:49.439659 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:49.439613 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b6ql\" (UniqueName: \"kubernetes.io/projected/2b65a77d-9939-4437-8737-8381937f9a53-kube-api-access-7b6ql\") pod \"test-curl-7081-00f201\" (UID: \"2b65a77d-9939-4437-8737-8381937f9a53\") " pod="opendatahub/test-curl-7081-00f201" Nov 03 20:22:49.558011 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:49.557692 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b6ql\" (UniqueName: \"kubernetes.io/projected/2b65a77d-9939-4437-8737-8381937f9a53-kube-api-access-7b6ql\") pod \"test-curl-7081-00f201\" (UID: \"2b65a77d-9939-4437-8737-8381937f9a53\") " pod="opendatahub/test-curl-7081-00f201" Nov 03 20:22:49.570128 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:49.570054 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b6ql\" (UniqueName: \"kubernetes.io/projected/2b65a77d-9939-4437-8737-8381937f9a53-kube-api-access-7b6ql\") pod \"test-curl-7081-00f201\" (UID: \"2b65a77d-9939-4437-8737-8381937f9a53\") " pod="opendatahub/test-curl-7081-00f201" Nov 03 20:22:49.647548 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:49.646257 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7081-00f201" Nov 03 20:22:49.989090 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:49.989058 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7081-00f201"] Nov 03 20:22:49.989821 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:22:49.989792 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b65a77d_9939_4437_8737_8381937f9a53.slice/crio-9244f1de1fe57a79b1243af4dd71c029dbd62782531e0f49d6b68a752bf906eb WatchSource:0}: Error finding container 9244f1de1fe57a79b1243af4dd71c029dbd62782531e0f49d6b68a752bf906eb: Status 404 returned error can't find the container with id 9244f1de1fe57a79b1243af4dd71c029dbd62782531e0f49d6b68a752bf906eb Nov 03 20:22:50.856205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:50.855722 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8" Nov 03 20:22:51.072733 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:51.072685 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7081-00f201" event={"ID":"2b65a77d-9939-4437-8737-8381937f9a53","Type":"ContainerStarted","Data":"8a9fa20585624866bf404f2a66d528fd0c49f0058c2b8cff716e6af864863428"} Nov 03 20:22:51.073001 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:51.072980 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7081-00f201" event={"ID":"2b65a77d-9939-4437-8737-8381937f9a53","Type":"ContainerStarted","Data":"9244f1de1fe57a79b1243af4dd71c029dbd62782531e0f49d6b68a752bf906eb"} Nov 03 20:22:51.118337 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:51.118098 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7081-00f201" podStartSLOduration=2.118082248 podStartE2EDuration="2.118082248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:51.107616105 +0000 UTC m=+1554.039483617" watchObservedRunningTime="2026-11-03 20:22:51.118082248 +0000 UTC m=+1554.049949758" Nov 03 20:22:51.937773 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:51.937727 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7081-00f201"] Nov 03 20:22:51.950228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:51.944135 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7081-00f201"] Nov 03 20:22:53.087664 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:53.087626 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7081-00f201" podUID="2b65a77d-9939-4437-8737-8381937f9a53" containerName="test-curl-7081-00f201" containerID="cri-o://8a9fa20585624866bf404f2a66d528fd0c49f0058c2b8cff716e6af864863428" gracePeriod=2 Nov 03 20:22:53.090796 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:22:53.090759 4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7081-00f201\" is forbidden: User \"system:node:ip-10-0-139-53.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.us-east-2.compute.internal' and this object" podUID="2b65a77d-9939-4437-8737-8381937f9a53" pod="opendatahub/test-curl-7081-00f201" Nov 03 20:22:54.891856 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:54.891826 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d" Nov 03 20:22:54.892316 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:54.891900 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2" Nov 03 20:22:55.328281 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:55.328244 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7081-00f201" Nov 03 20:22:55.435172 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:55.435137 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b65a77d-9939-4437-8737-8381937f9a53-kube-api-access-7b6ql\" (UniqueName: \"kubernetes.io/projected/2b65a77d-9939-4437-8737-8381937f9a53-kube-api-access-7b6ql\") pod \"2b65a77d-9939-4437-8737-8381937f9a53\" (UID: \"2b65a77d-9939-4437-8737-8381937f9a53\") " Nov 03 20:22:55.437342 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:55.437314 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b65a77d-9939-4437-8737-8381937f9a53-kube-api-access-7b6ql" pod "2b65a77d-9939-4437-8737-8381937f9a53" (UID: "2b65a77d-9939-4437-8737-8381937f9a53"). InnerVolumeSpecName "kube-api-access-7b6ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:55.537045 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:55.536969 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b6ql\" (UniqueName: \"kubernetes.io/projected/2b65a77d-9939-4437-8737-8381937f9a53-kube-api-access-7b6ql\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:56.112009 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:56.111972 4918 generic.go:356] "Generic (PLEG): container finished" podID="2b65a77d-9939-4437-8737-8381937f9a53" containerID="8a9fa20585624866bf404f2a66d528fd0c49f0058c2b8cff716e6af864863428" exitCode=137 Nov 03 20:22:56.112455 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:56.112026 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7081-00f201" Nov 03 20:22:56.112455 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:56.112064 4918 scope.go:122] "RemoveContainer" containerID="8a9fa20585624866bf404f2a66d528fd0c49f0058c2b8cff716e6af864863428" Nov 03 20:22:56.122824 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:56.122804 4918 scope.go:122] "RemoveContainer" containerID="8a9fa20585624866bf404f2a66d528fd0c49f0058c2b8cff716e6af864863428" Nov 03 20:22:56.123069 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:22:56.123048 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9fa20585624866bf404f2a66d528fd0c49f0058c2b8cff716e6af864863428\": container with ID starting with 8a9fa20585624866bf404f2a66d528fd0c49f0058c2b8cff716e6af864863428 not found: ID does not exist" containerID="8a9fa20585624866bf404f2a66d528fd0c49f0058c2b8cff716e6af864863428" Nov 03 20:22:56.123121 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:56.123077 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9fa20585624866bf404f2a66d528fd0c49f0058c2b8cff716e6af864863428"} err="failed to get container status \"8a9fa20585624866bf404f2a66d528fd0c49f0058c2b8cff716e6af864863428\": rpc error: code = NotFound desc = could not find container \"8a9fa20585624866bf404f2a66d528fd0c49f0058c2b8cff716e6af864863428\": container with ID starting with 8a9fa20585624866bf404f2a66d528fd0c49f0058c2b8cff716e6af864863428 not found: ID does not exist" Nov 03 20:22:56.123625 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:22:56.123606 4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7081-00f201\" is forbidden: User \"system:node:ip-10-0-139-53.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.us-east-2.compute.internal' and this object" podUID="2b65a77d-9939-4437-8737-8381937f9a53" pod="opendatahub/test-curl-7081-00f201" Nov 03 20:22:57.215002 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:22:57.214963 4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7081-00f201\" is forbidden: User \"system:node:ip-10-0-139-53.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.us-east-2.compute.internal' and this object" podUID="2b65a77d-9939-4437-8737-8381937f9a53" pod="opendatahub/test-curl-7081-00f201" Nov 03 20:22:57.217761 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:22:57.217730 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b65a77d-9939-4437-8737-8381937f9a53" path="/var/lib/kubelet/pods/2b65a77d-9939-4437-8737-8381937f9a53/volumes" Nov 03 20:22:58.209594 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:22:58.209557 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:23:17.940256 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:17.940001 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8"] Nov 03 20:23:28.223808 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:23:28.223762 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:23:33.023950 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.023894 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" podUID="f7b8c27b-baaa-4eb1-89b7-263cc25d7a08" containerName="main" containerID="cri-o://0c26b5088514a3476bed3e3fc29d9448300421fbcfbc03624f87f7211d3a6efb" gracePeriod=45 Nov 03 20:23:33.252873 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.252835 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv"] Nov 03 20:23:33.254393 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.254365 4918 cpu_manager.go:414] "containerMap: removing container" podUID="2b65a77d-9939-4437-8737-8381937f9a53" containerName="test-curl-7081-00f201" Nov 03 20:23:33.254393 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.254392 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b65a77d-9939-4437-8737-8381937f9a53" containerName="test-curl-7081-00f201" Nov 03 20:23:33.254609 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.254594 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b65a77d-9939-4437-8737-8381937f9a53" containerName="test-curl-7081-00f201" Nov 03 20:23:33.259449 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.259431 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv"] Nov 03 20:23:33.259651 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.259632 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.274473 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.274420 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-15e02e6b-openshift-default-dockercfg-mjzbc\"" Nov 03 20:23:33.327913 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.327884 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.328045 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.327938 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.328131 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.328105 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.328186 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.328155 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6hxx\" (UniqueName: \"kubernetes.io/projected/dd64d741-882d-4727-a3b3-cb9c09df011c-kube-api-access-b6hxx\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.329232 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.328507 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:23:33.329232 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.328582 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.329232 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.328702 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.329232 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.328742 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd64d741-882d-4727-a3b3-cb9c09df011c-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.329232 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.328959 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.329232 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.328991 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.429541 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.429510 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-home\" (UniqueName: \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-home\") pod \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " Nov 03 20:23:33.429745 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.429551 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-tls-certs\") pod \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " Nov 03 20:23:33.429745 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.429672 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-model-cache\") pod \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " Nov 03 20:23:33.429881 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.429742 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-kube-api-access-7m2lq\" (UniqueName: \"kubernetes.io/projected/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-kube-api-access-7m2lq\") pod \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " Nov 03 20:23:33.429881 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.429826 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-home" pod "f7b8c27b-baaa-4eb1-89b7-263cc25d7a08" (UID: "f7b8c27b-baaa-4eb1-89b7-263cc25d7a08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:33.429881 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.429844 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-tmp-dir\") pod \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " Nov 03 20:23:33.429881 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.429870 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-dshm\") pod \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\" (UID: \"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08\") " Nov 03 20:23:33.430094 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.429957 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-model-cache" pod "f7b8c27b-baaa-4eb1-89b7-263cc25d7a08" (UID: "f7b8c27b-baaa-4eb1-89b7-263cc25d7a08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:33.430094 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.430043 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-tmp-dir" pod "f7b8c27b-baaa-4eb1-89b7-263cc25d7a08" (UID: "f7b8c27b-baaa-4eb1-89b7-263cc25d7a08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:33.430342 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.430315 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.430482 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.430362 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6hxx\" (UniqueName: \"kubernetes.io/projected/dd64d741-882d-4727-a3b3-cb9c09df011c-kube-api-access-b6hxx\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.430734 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.430713 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.431038 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.431016 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.431251 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.431114 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.431382 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.431225 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd64d741-882d-4727-a3b3-cb9c09df011c-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.431632 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.431610 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.431747 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.431732 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.431869 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.431853 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.431983 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.431970 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.432202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.432169 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-model-cache\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:33.432285 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.432207 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-tmp-dir\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:33.432285 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.432221 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-home\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:33.432398 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.432291 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd64d741-882d-4727-a3b3-cb9c09df011c-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.432508 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.432488 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.432745 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.432708 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.432810 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.432779 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.433806 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.433783 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.433931 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.433908 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-tls-certs" pod "f7b8c27b-baaa-4eb1-89b7-263cc25d7a08" (UID: "f7b8c27b-baaa-4eb1-89b7-263cc25d7a08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:23:33.434001 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.433921 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-dshm" pod "f7b8c27b-baaa-4eb1-89b7-263cc25d7a08" (UID: "f7b8c27b-baaa-4eb1-89b7-263cc25d7a08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:33.434001 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.433936 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-kube-api-access-7m2lq" pod "f7b8c27b-baaa-4eb1-89b7-263cc25d7a08" (UID: "f7b8c27b-baaa-4eb1-89b7-263cc25d7a08"). InnerVolumeSpecName "kube-api-access-7m2lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:33.434156 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.434065 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.511457 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.511418 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.511735 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.511693 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6hxx\" (UniqueName: \"kubernetes.io/projected/dd64d741-882d-4727-a3b3-cb9c09df011c-kube-api-access-b6hxx\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.534289 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.534208 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-dshm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:33.534289 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.534244 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-tls-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:33.534289 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.534261 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m2lq\" (UniqueName: \"kubernetes.io/projected/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08-kube-api-access-7m2lq\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:33.581757 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.581718 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:33.692755 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.692723 4918 generic.go:356] "Generic (PLEG): container finished" podID="f7b8c27b-baaa-4eb1-89b7-263cc25d7a08" containerID="0c26b5088514a3476bed3e3fc29d9448300421fbcfbc03624f87f7211d3a6efb" exitCode=0 Nov 03 20:23:33.692926 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.692801 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" Nov 03 20:23:33.692926 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.692812 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" event={"ID":"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08","Type":"ContainerDied","Data":"0c26b5088514a3476bed3e3fc29d9448300421fbcfbc03624f87f7211d3a6efb"} Nov 03 20:23:33.692926 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.692869 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8" event={"ID":"f7b8c27b-baaa-4eb1-89b7-263cc25d7a08","Type":"ContainerDied","Data":"7d5ab9d05f1c72ef0de817d433eb2a2af4e5b5bcc49a065ac09c0c9f9fbd6787"} Nov 03 20:23:33.692926 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.692894 4918 scope.go:122] "RemoveContainer" containerID="0c26b5088514a3476bed3e3fc29d9448300421fbcfbc03624f87f7211d3a6efb" Nov 03 20:23:33.713616 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.713594 4918 scope.go:122] "RemoveContainer" containerID="0c26b5088514a3476bed3e3fc29d9448300421fbcfbc03624f87f7211d3a6efb" Nov 03 20:23:33.713952 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:23:33.713930 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c26b5088514a3476bed3e3fc29d9448300421fbcfbc03624f87f7211d3a6efb\": container with ID starting with 0c26b5088514a3476bed3e3fc29d9448300421fbcfbc03624f87f7211d3a6efb not found: ID does not exist" containerID="0c26b5088514a3476bed3e3fc29d9448300421fbcfbc03624f87f7211d3a6efb" Nov 03 20:23:33.714042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.713962 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c26b5088514a3476bed3e3fc29d9448300421fbcfbc03624f87f7211d3a6efb"} err="failed to get container status \"0c26b5088514a3476bed3e3fc29d9448300421fbcfbc03624f87f7211d3a6efb\": rpc error: code = NotFound desc = could not find container \"0c26b5088514a3476bed3e3fc29d9448300421fbcfbc03624f87f7211d3a6efb\": container with ID starting with 0c26b5088514a3476bed3e3fc29d9448300421fbcfbc03624f87f7211d3a6efb not found: ID does not exist" Nov 03 20:23:33.736784 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.736753 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8"] Nov 03 20:23:33.742732 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.742693 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt"] Nov 03 20:23:33.744104 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.744077 4918 cpu_manager.go:414] "containerMap: removing container" podUID="f7b8c27b-baaa-4eb1-89b7-263cc25d7a08" containerName="main" Nov 03 20:23:33.744261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.744245 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b8c27b-baaa-4eb1-89b7-263cc25d7a08" containerName="main" Nov 03 20:23:33.744635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.744615 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b8c27b-baaa-4eb1-89b7-263cc25d7a08" containerName="main" Nov 03 20:23:33.786892 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.786420 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt" Nov 03 20:23:33.814973 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.808466 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d9f082-dockercfg-mlzvj\"" Nov 03 20:23:33.816301 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.816270 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-15e02e6b/e2e-lifecycle-model-15e02e6b-kserve-647cb8969c-npgp8"] Nov 03 20:23:33.816718 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.816700 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d9f082\"" Nov 03 20:23:33.823707 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:33.823680 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt"] Nov 03 20:23:34.093844 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.080029 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:23:34.093844 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.080124 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:23:34.093844 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.080171 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:23:34.135383 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.108711 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-487pv\" (UniqueName: \"kubernetes.io/projected/b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f-kube-api-access-487pv\") pod \"payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt\" (UID: \"b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt" Nov 03 20:23:34.135383 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.108966 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt\" (UID: \"b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt" Nov 03 20:23:34.168432 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.168401 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv"] Nov 03 20:23:34.175348 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.175316 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd"] Nov 03 20:23:34.198398 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.198367 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd"] Nov 03 20:23:34.198632 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.198605 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:34.208685 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.208661 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6cf8ae\"/\"openshift-service-ca.crt\"" Nov 03 20:23:34.208990 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.208968 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6cf8ae\"/\"default-dockercfg-5gn4c\"" Nov 03 20:23:34.209258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.209238 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6cf8ae\"/\"test-model-6cf8ae-kserve-self-signed-certs\"" Nov 03 20:23:34.209513 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.209493 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6cf8ae\"/\"kube-root-ca.crt\"" Nov 03 20:23:34.210704 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.210676 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt\" (UID: \"b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt" Nov 03 20:23:34.211083 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.211051 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-487pv\" (UniqueName: \"kubernetes.io/projected/b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f-kube-api-access-487pv\") pod \"payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt\" (UID: \"b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt" Nov 03 20:23:34.212360 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.212333 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt\" (UID: \"b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt" Nov 03 20:23:34.224065 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.224041 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-487pv\" (UniqueName: \"kubernetes.io/projected/b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f-kube-api-access-487pv\") pod \"payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt\" (UID: \"b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt" Nov 03 20:23:34.312428 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.312380 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-dshm\") pod \"test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:34.312627 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.312435 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-model-cache\") pod \"test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:34.312700 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.312624 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-tmp-dir\") pod \"test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:34.312700 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.312664 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-tls-certs\") pod \"test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:34.312812 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.312788 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-home\") pod \"test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:34.312956 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.312935 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp8q9\" (UniqueName: \"kubernetes.io/projected/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-kube-api-access-xp8q9\") pod \"test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:34.389255 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.372478 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt" Nov 03 20:23:34.472668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.470406 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp8q9\" (UniqueName: \"kubernetes.io/projected/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-kube-api-access-xp8q9\") pod \"test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:34.472668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.470583 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-dshm\") pod \"test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:34.472668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.470616 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-model-cache\") pod \"test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:34.472668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.470748 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-tmp-dir\") pod \"test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:34.472668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.470781 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-tls-certs\") pod \"test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:34.472668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.470913 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-home\") pod \"test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:34.472668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.471407 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-home\") pod \"test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:34.472668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.471422 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-model-cache\") pod \"test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:34.472668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.471642 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-tmp-dir\") pod \"test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:34.492204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.491732 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-tls-certs\") pod \"test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:34.492204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.492019 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-dshm\") pod \"test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:34.494385 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.494338 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp8q9\" (UniqueName: \"kubernetes.io/projected/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-kube-api-access-xp8q9\") pod \"test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:34.564125 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.561950 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:34.738783 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.737464 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" event={"ID":"dd64d741-882d-4727-a3b3-cb9c09df011c","Type":"ContainerStarted","Data":"2fe2d628c4b6ad5720b2c7ab434b8a423e98531036306433409cc182ce702542"} Nov 03 20:23:34.738783 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.737533 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" event={"ID":"dd64d741-882d-4727-a3b3-cb9c09df011c","Type":"ContainerStarted","Data":"4580983d3e33f47e7c1d9bd1832684496e2f87b02693c9ba6ec8d3d3670f0aab"} Nov 03 20:23:34.778221 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.768626 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" podStartSLOduration=1.7686065279999998 podStartE2EDuration="1.768606528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:34.765502236 +0000 UTC m=+1597.697369772" watchObservedRunningTime="2026-11-03 20:23:34.768606528 +0000 UTC m=+1597.700474042" Nov 03 20:23:34.871752 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.871713 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt"] Nov 03 20:23:34.874218 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:23:34.874172 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ac3141_ef2a_4b03_9c26_8cc5cc36a12f.slice/crio-e34aa25bedc73dccca4df66c0c0e25cd3a181058cb0fd85abbb7e7590d67f7b7 WatchSource:0}: Error finding container e34aa25bedc73dccca4df66c0c0e25cd3a181058cb0fd85abbb7e7590d67f7b7: Status 404 returned error can't find the container with id e34aa25bedc73dccca4df66c0c0e25cd3a181058cb0fd85abbb7e7590d67f7b7 Nov 03 20:23:34.951200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:34.950265 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd"] Nov 03 20:23:34.965399 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:23:34.965365 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8de9e587_15c4_48b3_bcfe_ac1e7a2b0fa0.slice/crio-8cf5b1a6e3e008be90db1e61ea2f351fd0615af616ec55dead297db2bd79d0a8 WatchSource:0}: Error finding container 8cf5b1a6e3e008be90db1e61ea2f351fd0615af616ec55dead297db2bd79d0a8: Status 404 returned error can't find the container with id 8cf5b1a6e3e008be90db1e61ea2f351fd0615af616ec55dead297db2bd79d0a8 Nov 03 20:23:35.236982 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:35.236887 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b8c27b-baaa-4eb1-89b7-263cc25d7a08" path="/var/lib/kubelet/pods/f7b8c27b-baaa-4eb1-89b7-263cc25d7a08/volumes" Nov 03 20:23:35.582368 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:35.581912 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:35.587944 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:35.587860 4918 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:23:35.587944 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:35.587923 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" podUID="dd64d741-882d-4727-a3b3-cb9c09df011c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:23:35.751920 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:35.751879 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt" event={"ID":"b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f","Type":"ContainerStarted","Data":"ac9e9de6f4bf9505f9481047a3da48824dd080196c8bd9beb364e0bb2f971dc6"} Nov 03 20:23:35.752088 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:35.751954 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt" Nov 03 20:23:35.752088 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:35.751979 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt" event={"ID":"b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f","Type":"ContainerStarted","Data":"e34aa25bedc73dccca4df66c0c0e25cd3a181058cb0fd85abbb7e7590d67f7b7"} Nov 03 20:23:35.753782 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:35.753748 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" event={"ID":"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0","Type":"ContainerStarted","Data":"489e10e04f01626abd2903685fd26fcb7353ac44183c963f337d9bb4e01a47ce"} Nov 03 20:23:35.753905 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:35.753797 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" event={"ID":"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0","Type":"ContainerStarted","Data":"8cf5b1a6e3e008be90db1e61ea2f351fd0615af616ec55dead297db2bd79d0a8"} Nov 03 20:23:35.771528 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:35.771484 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt" podStartSLOduration=2.771470775 podStartE2EDuration="2.771470775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:35.770378455 +0000 UTC m=+1598.702245967" watchObservedRunningTime="2026-11-03 20:23:35.771470775 +0000 UTC m=+1598.703338286" Nov 03 20:23:35.797414 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:35.794087 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" podStartSLOduration=1.794069371 podStartE2EDuration="1.794069371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:35.789222116 +0000 UTC m=+1598.721089628" watchObservedRunningTime="2026-11-03 20:23:35.794069371 +0000 UTC m=+1598.725936883" Nov 03 20:23:36.573229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:36.572138 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:36.582292 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:36.582255 4918 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:23:36.582438 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:36.582327 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" podUID="dd64d741-882d-4727-a3b3-cb9c09df011c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:23:37.583064 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:37.583027 4918 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:23:37.583488 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:37.583095 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" podUID="dd64d741-882d-4727-a3b3-cb9c09df011c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:23:38.582894 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:38.582854 4918 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:23:38.583108 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:38.582911 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" podUID="dd64d741-882d-4727-a3b3-cb9c09df011c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:23:39.582347 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:39.582311 4918 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:23:39.582518 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:39.582380 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" podUID="dd64d741-882d-4727-a3b3-cb9c09df011c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:23:40.591352 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:40.590205 4918 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:23:40.591352 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:40.590290 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" podUID="dd64d741-882d-4727-a3b3-cb9c09df011c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:23:41.587424 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:41.587391 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:41.587903 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:41.587879 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:41.588779 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:41.588757 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:23:44.585200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:44.581078 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:44.608255 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:44.603554 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:44.891714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:44.891639 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:23:46.766351 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:46.766320 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt" Nov 03 20:23:48.275586 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.275540 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp"] Nov 03 20:23:48.283336 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.283308 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.300023 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.299993 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp"] Nov 03 20:23:48.419102 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.419061 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.419308 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.419122 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.419308 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.419269 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.419383 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.419331 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7lc\" (UniqueName: \"kubernetes.io/projected/993f0b44-a09e-44de-b098-d1c973c51d1b-kube-api-access-qn7lc\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.419510 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.419494 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/993f0b44-a09e-44de-b098-d1c973c51d1b-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.419552 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.419537 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.419652 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.419636 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.419684 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.419669 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.419718 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.419688 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.520931 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.520893 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.521125 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.520954 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.521125 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.520985 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.521125 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.521042 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7lc\" (UniqueName: \"kubernetes.io/projected/993f0b44-a09e-44de-b098-d1c973c51d1b-kube-api-access-qn7lc\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.521343 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.521201 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/993f0b44-a09e-44de-b098-d1c973c51d1b-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.521343 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.521261 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.521343 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.521338 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.521511 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.521375 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.521511 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.521379 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.521511 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.521400 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.521511 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.521451 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.521753 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.521729 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.521871 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.521851 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.521962 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.521940 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/993f0b44-a09e-44de-b098-d1c973c51d1b-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.523749 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.523727 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.523959 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.523941 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.533039 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.532988 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7lc\" (UniqueName: \"kubernetes.io/projected/993f0b44-a09e-44de-b098-d1c973c51d1b-kube-api-access-qn7lc\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.536635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.536613 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.602789 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.602752 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:48.898722 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:23:48.898681 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993f0b44_a09e_44de_b098_d1c973c51d1b.slice/crio-adb99f08d8a8c160aebfda9e1d770e085b2c2b3c81dc538af30cf1093a310c92 WatchSource:0}: Error finding container adb99f08d8a8c160aebfda9e1d770e085b2c2b3c81dc538af30cf1093a310c92: Status 404 returned error can't find the container with id adb99f08d8a8c160aebfda9e1d770e085b2c2b3c81dc538af30cf1093a310c92 Nov 03 20:23:48.899025 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.898998 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp"] Nov 03 20:23:48.901904 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.901863 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:23:48.902017 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.901937 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:23:48.902017 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.901965 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:23:48.912062 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:48.912028 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" event={"ID":"993f0b44-a09e-44de-b098-d1c973c51d1b","Type":"ContainerStarted","Data":"adb99f08d8a8c160aebfda9e1d770e085b2c2b3c81dc538af30cf1093a310c92"} Nov 03 20:23:49.926281 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:49.926240 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" event={"ID":"993f0b44-a09e-44de-b098-d1c973c51d1b","Type":"ContainerStarted","Data":"3a46c40647598bf40de4d2cd35856e48f50564768d13f5a498b892d0ab6a14a7"} Nov 03 20:23:49.992197 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:49.990248 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" podStartSLOduration=1.990226506 podStartE2EDuration="1.990226506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:49.965010755 +0000 UTC m=+1612.896878292" watchObservedRunningTime="2026-11-03 20:23:49.990226506 +0000 UTC m=+1612.922094017" Nov 03 20:23:50.603678 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:50.603645 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:50.605356 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:50.605328 4918 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 20:23:50.605473 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:50.605378 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" podUID="993f0b44-a09e-44de-b098-d1c973c51d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 20:23:51.603437 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:51.603402 4918 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 20:23:51.603929 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:51.603481 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" podUID="993f0b44-a09e-44de-b098-d1c973c51d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 20:23:52.457200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.456225 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt"] Nov 03 20:23:52.462689 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.462520 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt"] Nov 03 20:23:52.462689 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.462624 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:52.471096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.471060 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5ca474\"/\"kube-root-ca.crt\"" Nov 03 20:23:52.472124 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.472097 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5ca474\"/\"test-model-5ca474-kserve-self-signed-certs\"" Nov 03 20:23:52.472316 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.472194 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5ca474\"/\"openshift-service-ca.crt\"" Nov 03 20:23:52.472386 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.472372 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5ca474\"/\"default-dockercfg-54rxs\"" Nov 03 20:23:52.551501 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.551465 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-home\") pod \"test-model-5ca474-kserve-5b7954d967-8hfgt\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:52.551736 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.551508 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxshx\" (UniqueName: \"kubernetes.io/projected/97238908-155f-4656-8767-fae2a876d653-kube-api-access-zxshx\") pod \"test-model-5ca474-kserve-5b7954d967-8hfgt\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:52.551736 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.551613 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97238908-155f-4656-8767-fae2a876d653-tls-certs\") pod \"test-model-5ca474-kserve-5b7954d967-8hfgt\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:52.551736 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.551684 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-tmp-dir\") pod \"test-model-5ca474-kserve-5b7954d967-8hfgt\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:52.551736 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.551735 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-model-cache\") pod \"test-model-5ca474-kserve-5b7954d967-8hfgt\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:52.552079 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.551814 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-dshm\") pod \"test-model-5ca474-kserve-5b7954d967-8hfgt\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:52.603605 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.603570 4918 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 20:23:52.603979 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.603641 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" podUID="993f0b44-a09e-44de-b098-d1c973c51d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 20:23:52.653397 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.653349 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97238908-155f-4656-8767-fae2a876d653-tls-certs\") pod \"test-model-5ca474-kserve-5b7954d967-8hfgt\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:52.653594 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.653464 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-tmp-dir\") pod \"test-model-5ca474-kserve-5b7954d967-8hfgt\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:52.653594 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.653516 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-model-cache\") pod \"test-model-5ca474-kserve-5b7954d967-8hfgt\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:52.653677 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.653618 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-dshm\") pod \"test-model-5ca474-kserve-5b7954d967-8hfgt\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:52.654055 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.654024 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-home\") pod \"test-model-5ca474-kserve-5b7954d967-8hfgt\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:52.654055 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.654044 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-tmp-dir\") pod \"test-model-5ca474-kserve-5b7954d967-8hfgt\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:52.654277 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.654081 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxshx\" (UniqueName: \"kubernetes.io/projected/97238908-155f-4656-8767-fae2a876d653-kube-api-access-zxshx\") pod \"test-model-5ca474-kserve-5b7954d967-8hfgt\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:52.654277 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.654083 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-model-cache\") pod \"test-model-5ca474-kserve-5b7954d967-8hfgt\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:52.654358 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.654315 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-home\") pod \"test-model-5ca474-kserve-5b7954d967-8hfgt\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:52.656362 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.656332 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-dshm\") pod \"test-model-5ca474-kserve-5b7954d967-8hfgt\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:52.656744 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.656722 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97238908-155f-4656-8767-fae2a876d653-tls-certs\") pod \"test-model-5ca474-kserve-5b7954d967-8hfgt\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:52.666305 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.666272 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxshx\" (UniqueName: \"kubernetes.io/projected/97238908-155f-4656-8767-fae2a876d653-kube-api-access-zxshx\") pod \"test-model-5ca474-kserve-5b7954d967-8hfgt\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:52.800248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:52.797344 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:53.462200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:53.461222 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt"] Nov 03 20:23:53.608348 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:53.608312 4918 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 20:23:53.608815 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:53.608376 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" podUID="993f0b44-a09e-44de-b098-d1c973c51d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 20:23:54.000004 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:53.999905 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" event={"ID":"97238908-155f-4656-8767-fae2a876d653","Type":"ContainerStarted","Data":"d6b8b9d717059d83f58b8196843dde345caf04f005df518a62bae98cb105637d"} Nov 03 20:23:54.000004 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:53.999963 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" event={"ID":"97238908-155f-4656-8767-fae2a876d653","Type":"ContainerStarted","Data":"1e6c26049fd6dfdd9a2350437c0e9cc78989b7242cb5a7be0b7838f4a6142f8c"} Nov 03 20:23:54.052308 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:54.052249 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" podStartSLOduration=2.05223363 podStartE2EDuration="2.05223363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:54.049646692 +0000 UTC m=+1616.981514204" watchObservedRunningTime="2026-11-03 20:23:54.05223363 +0000 UTC m=+1616.984101140" Nov 03 20:23:54.631646 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:54.631607 4918 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 20:23:54.649691 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:54.649639 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" podUID="993f0b44-a09e-44de-b098-d1c973c51d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 20:23:54.798706 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:54.798623 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:23:55.603443 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:55.603407 4918 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 20:23:55.603623 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:55.603472 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" podUID="993f0b44-a09e-44de-b098-d1c973c51d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 20:23:56.608498 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:56.608464 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:56.609837 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:56.609805 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:56.610624 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:23:56.610605 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:23:58.223806 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:23:58.223762 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:24:02.797856 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:02.797817 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:24:02.810674 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:02.810645 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:24:03.328224 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.328150 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj"] Nov 03 20:24:03.334495 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.334461 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.339782 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.339754 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4"] Nov 03 20:24:03.348635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.348608 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj"] Nov 03 20:24:03.348764 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.348704 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.377322 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.377289 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4"] Nov 03 20:24:03.412630 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.412597 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.412630 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.412637 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.412875 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.412724 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.412875 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.412760 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.412875 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.412782 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.413003 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.412891 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.413003 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.412927 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.413003 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.412991 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148d938c-d92c-454a-833d-4d2b9b69d9ff-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.413155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.413031 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.413155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.413057 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.413155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.413104 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkk9m\" (UniqueName: \"kubernetes.io/projected/148d938c-d92c-454a-833d-4d2b9b69d9ff-kube-api-access-jkk9m\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.413155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.413134 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.413448 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.413198 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.413448 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.413295 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgrwj\" (UniqueName: \"kubernetes.io/projected/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-kube-api-access-sgrwj\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.413448 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.413368 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.413448 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.413394 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.413448 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.413429 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.413711 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.413470 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.414391 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.414362 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:24:03.514556 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.514521 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.514724 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.514583 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148d938c-d92c-454a-833d-4d2b9b69d9ff-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.514724 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.514615 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.514724 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.514635 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.514898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.514779 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkk9m\" (UniqueName: \"kubernetes.io/projected/148d938c-d92c-454a-833d-4d2b9b69d9ff-kube-api-access-jkk9m\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.514898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.514824 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.514898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.514869 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.515014 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.514964 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgrwj\" (UniqueName: \"kubernetes.io/projected/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-kube-api-access-sgrwj\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.515066 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.515035 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.515066 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.515048 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.515168 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.515063 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.515168 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.515121 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.515303 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.515173 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.515303 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.515249 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.515303 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.515258 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.515456 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.515320 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.515456 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.515357 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.515456 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.515410 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.515456 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.515415 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.515669 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.515474 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.515669 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.515508 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.515669 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.515582 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.515837 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.515711 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.515837 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.515824 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.515944 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.515893 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.516027 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.516006 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.516093 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.516028 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.516587 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.516558 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148d938c-d92c-454a-833d-4d2b9b69d9ff-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.517998 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.517968 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.518121 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.518062 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.518325 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.518200 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.519120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.519097 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.527207 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.527169 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.527326 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.527315 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgrwj\" (UniqueName: \"kubernetes.io/projected/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-kube-api-access-sgrwj\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.527398 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.527363 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.527730 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.527706 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkk9m\" (UniqueName: \"kubernetes.io/projected/148d938c-d92c-454a-833d-4d2b9b69d9ff-kube-api-access-jkk9m\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:03.658833 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.658738 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:03.696613 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:03.696578 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:04.000695 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:24:04.000665 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df7a75c_bc28_4e1c_989a_85b9fe3143c7.slice/crio-a24c5cc1babc51d6feab592261db7800df1df7c2dd3ee9f86ca22c237dd6be96 WatchSource:0}: Error finding container a24c5cc1babc51d6feab592261db7800df1df7c2dd3ee9f86ca22c237dd6be96: Status 404 returned error can't find the container with id a24c5cc1babc51d6feab592261db7800df1df7c2dd3ee9f86ca22c237dd6be96 Nov 03 20:24:04.001585 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.001558 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj"] Nov 03 20:24:04.003723 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.003692 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:24:04.003793 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.003755 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:24:04.003793 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.003784 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:24:04.067601 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:24:04.067576 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148d938c_d92c_454a_833d_4d2b9b69d9ff.slice/crio-d88d1bd1750098677d87025cf47b9292591de665da944e925e3fe92dcc5b5e88 WatchSource:0}: Error finding container d88d1bd1750098677d87025cf47b9292591de665da944e925e3fe92dcc5b5e88: Status 404 returned error can't find the container with id d88d1bd1750098677d87025cf47b9292591de665da944e925e3fe92dcc5b5e88 Nov 03 20:24:04.069280 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.069251 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4"] Nov 03 20:24:04.070803 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.070765 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:24:04.070886 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.070836 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:24:04.070886 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.070864 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:24:04.407426 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.407385 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" event={"ID":"148d938c-d92c-454a-833d-4d2b9b69d9ff","Type":"ContainerStarted","Data":"fa376c2a180c62326d72a1a669a62600221259a91e5f1b15b013337e4a4d566b"} Nov 03 20:24:04.407628 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.407448 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" event={"ID":"148d938c-d92c-454a-833d-4d2b9b69d9ff","Type":"ContainerStarted","Data":"d88d1bd1750098677d87025cf47b9292591de665da944e925e3fe92dcc5b5e88"} Nov 03 20:24:04.410152 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.410126 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" event={"ID":"3df7a75c-bc28-4e1c-989a-85b9fe3143c7","Type":"ContainerStarted","Data":"2c2389939731f479f2fa10bfcddc1251bd88addfe681114ff22f8378989e2adb"} Nov 03 20:24:04.410297 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.410167 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" event={"ID":"3df7a75c-bc28-4e1c-989a-85b9fe3143c7","Type":"ContainerStarted","Data":"a24c5cc1babc51d6feab592261db7800df1df7c2dd3ee9f86ca22c237dd6be96"} Nov 03 20:24:04.437724 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.437661 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" podStartSLOduration=1.437642808 podStartE2EDuration="1.437642808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:04.431104294 +0000 UTC m=+1627.362971806" watchObservedRunningTime="2026-11-03 20:24:04.437642808 +0000 UTC m=+1627.369510320" Nov 03 20:24:04.468578 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.468517 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" podStartSLOduration=1.468503831 podStartE2EDuration="1.468503831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:04.462005532 +0000 UTC m=+1627.393873044" watchObservedRunningTime="2026-11-03 20:24:04.468503831 +0000 UTC m=+1627.400371409" Nov 03 20:24:04.659936 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.659857 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:04.696835 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.696803 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:04.862204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.861610 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj"] Nov 03 20:24:04.869572 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.869454 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv"] Nov 03 20:24:04.869756 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.869698 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" podUID="dd64d741-882d-4727-a3b3-cb9c09df011c" containerName="istio-proxy" containerID="cri-o://2fe2d628c4b6ad5720b2c7ab434b8a423e98531036306433409cc182ce702542" gracePeriod=30 Nov 03 20:24:04.893654 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.887016 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp"] Nov 03 20:24:04.893654 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.887298 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" podUID="993f0b44-a09e-44de-b098-d1c973c51d1b" containerName="istio-proxy" containerID="cri-o://3a46c40647598bf40de4d2cd35856e48f50564768d13f5a498b892d0ab6a14a7" gracePeriod=30 Nov 03 20:24:04.893654 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:04.892784 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4"] Nov 03 20:24:05.421726 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:05.421695 4918 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" secret="" err="secret \"e2e-mt-15e02e6b-openshift-default-dockercfg-mjzbc\" not found" Nov 03 20:24:05.422165 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:05.421914 4918 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" secret="" err="secret \"e2e-mt-15e02e6b-openshift-default-dockercfg-mjzbc\" not found" Nov 03 20:24:05.664914 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:05.664884 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:05.704415 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:05.704338 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:06.451788 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:06.451743 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" podUID="148d938c-d92c-454a-833d-4d2b9b69d9ff" containerName="istio-proxy" containerID="cri-o://fa376c2a180c62326d72a1a669a62600221259a91e5f1b15b013337e4a4d566b" gracePeriod=30 Nov 03 20:24:06.465945 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:06.465837 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:06.466431 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:06.466384 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" podUID="3df7a75c-bc28-4e1c-989a-85b9fe3143c7" containerName="istio-proxy" containerID="cri-o://2c2389939731f479f2fa10bfcddc1251bd88addfe681114ff22f8378989e2adb" gracePeriod=30 Nov 03 20:24:06.468878 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:06.468853 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:06.492469 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:06.480653 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:06.492469 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:06.481069 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:10.219772 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.219740 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:24:10.264595 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.264562 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:24:10.320829 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.320797 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-podinfo\") pod \"dd64d741-882d-4727-a3b3-cb9c09df011c\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " Nov 03 20:24:10.320985 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.320834 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-token\" (UniqueName: \"kubernetes.io/projected/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-token\") pod \"dd64d741-882d-4727-a3b3-cb9c09df011c\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " Nov 03 20:24:10.320985 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.320945 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-workload-socket\") pod \"993f0b44-a09e-44de-b098-d1c973c51d1b\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " Nov 03 20:24:10.320985 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.320968 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd64d741-882d-4727-a3b3-cb9c09df011c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd64d741-882d-4727-a3b3-cb9c09df011c-istiod-ca-cert\") pod \"dd64d741-882d-4727-a3b3-cb9c09df011c\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " Nov 03 20:24:10.321106 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.321032 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-workload-socket\") pod \"dd64d741-882d-4727-a3b3-cb9c09df011c\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " Nov 03 20:24:10.321106 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.321072 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-credential-socket\") pod \"dd64d741-882d-4727-a3b3-cb9c09df011c\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " Nov 03 20:24:10.321196 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.321117 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-token\" (UniqueName: \"kubernetes.io/projected/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-token\") pod \"993f0b44-a09e-44de-b098-d1c973c51d1b\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " Nov 03 20:24:10.321262 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.321233 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-envoy\") pod \"993f0b44-a09e-44de-b098-d1c973c51d1b\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " Nov 03 20:24:10.321334 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.321291 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-workload-socket" pod "993f0b44-a09e-44de-b098-d1c973c51d1b" (UID: "993f0b44-a09e-44de-b098-d1c973c51d1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.321334 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.321307 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993f0b44-a09e-44de-b098-d1c973c51d1b-kube-api-access-qn7lc\" (UniqueName: \"kubernetes.io/projected/993f0b44-a09e-44de-b098-d1c973c51d1b-kube-api-access-qn7lc\") pod \"993f0b44-a09e-44de-b098-d1c973c51d1b\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " Nov 03 20:24:10.321438 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.321342 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-credential-socket\") pod \"993f0b44-a09e-44de-b098-d1c973c51d1b\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " Nov 03 20:24:10.321438 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.321386 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-envoy\") pod \"dd64d741-882d-4727-a3b3-cb9c09df011c\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " Nov 03 20:24:10.321589 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.321559 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-data\") pod \"dd64d741-882d-4727-a3b3-cb9c09df011c\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " Nov 03 20:24:10.321669 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.321574 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-credential-socket" pod "dd64d741-882d-4727-a3b3-cb9c09df011c" (UID: "dd64d741-882d-4727-a3b3-cb9c09df011c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.321669 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.321608 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-data\") pod \"993f0b44-a09e-44de-b098-d1c973c51d1b\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " Nov 03 20:24:10.321669 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.321656 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993f0b44-a09e-44de-b098-d1c973c51d1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/993f0b44-a09e-44de-b098-d1c973c51d1b-istiod-ca-cert\") pod \"993f0b44-a09e-44de-b098-d1c973c51d1b\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " Nov 03 20:24:10.321834 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.321684 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-podinfo\") pod \"993f0b44-a09e-44de-b098-d1c973c51d1b\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " Nov 03 20:24:10.321834 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.321813 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-workload-certs\") pod \"993f0b44-a09e-44de-b098-d1c973c51d1b\" (UID: \"993f0b44-a09e-44de-b098-d1c973c51d1b\") " Nov 03 20:24:10.321966 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.321927 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd64d741-882d-4727-a3b3-cb9c09df011c-istiod-ca-cert" pod "dd64d741-882d-4727-a3b3-cb9c09df011c" (UID: "dd64d741-882d-4727-a3b3-cb9c09df011c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:10.321966 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.321955 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-workload-certs\") pod \"dd64d741-882d-4727-a3b3-cb9c09df011c\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " Nov 03 20:24:10.322090 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.321973 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-credential-socket" pod "993f0b44-a09e-44de-b098-d1c973c51d1b" (UID: "993f0b44-a09e-44de-b098-d1c973c51d1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.322090 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.322026 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd64d741-882d-4727-a3b3-cb9c09df011c-kube-api-access-b6hxx\" (UniqueName: \"kubernetes.io/projected/dd64d741-882d-4727-a3b3-cb9c09df011c-kube-api-access-b6hxx\") pod \"dd64d741-882d-4727-a3b3-cb9c09df011c\" (UID: \"dd64d741-882d-4727-a3b3-cb9c09df011c\") " Nov 03 20:24:10.323411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.322305 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-workload-socket" pod "dd64d741-882d-4727-a3b3-cb9c09df011c" (UID: "dd64d741-882d-4727-a3b3-cb9c09df011c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.323411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.322515 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993f0b44-a09e-44de-b098-d1c973c51d1b-istiod-ca-cert" pod "993f0b44-a09e-44de-b098-d1c973c51d1b" (UID: "993f0b44-a09e-44de-b098-d1c973c51d1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:10.323411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.322722 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-workload-certs" pod "993f0b44-a09e-44de-b098-d1c973c51d1b" (UID: "993f0b44-a09e-44de-b098-d1c973c51d1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.323411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.323337 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.323411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.323362 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd64d741-882d-4727-a3b3-cb9c09df011c-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.323411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.323376 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.323411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.323394 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.323411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.323406 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.323892 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.323421 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/993f0b44-a09e-44de-b098-d1c973c51d1b-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.323892 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.323435 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.323892 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.323563 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-data" pod "dd64d741-882d-4727-a3b3-cb9c09df011c" (UID: "dd64d741-882d-4727-a3b3-cb9c09df011c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.323892 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.323665 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-data" pod "993f0b44-a09e-44de-b098-d1c973c51d1b" (UID: "993f0b44-a09e-44de-b098-d1c973c51d1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.323892 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.323781 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-workload-certs" pod "dd64d741-882d-4727-a3b3-cb9c09df011c" (UID: "dd64d741-882d-4727-a3b3-cb9c09df011c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.324607 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.324570 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-token" pod "dd64d741-882d-4727-a3b3-cb9c09df011c" (UID: "dd64d741-882d-4727-a3b3-cb9c09df011c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:10.324761 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.324733 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-podinfo" pod "dd64d741-882d-4727-a3b3-cb9c09df011c" (UID: "dd64d741-882d-4727-a3b3-cb9c09df011c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:24:10.324872 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.324847 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd64d741-882d-4727-a3b3-cb9c09df011c-kube-api-access-b6hxx" pod "dd64d741-882d-4727-a3b3-cb9c09df011c" (UID: "dd64d741-882d-4727-a3b3-cb9c09df011c"). InnerVolumeSpecName "kube-api-access-b6hxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:10.325303 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.325240 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-token" pod "993f0b44-a09e-44de-b098-d1c973c51d1b" (UID: "993f0b44-a09e-44de-b098-d1c973c51d1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:10.325399 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.325334 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-envoy" pod "993f0b44-a09e-44de-b098-d1c973c51d1b" (UID: "993f0b44-a09e-44de-b098-d1c973c51d1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.325776 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.325752 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-envoy" pod "dd64d741-882d-4727-a3b3-cb9c09df011c" (UID: "dd64d741-882d-4727-a3b3-cb9c09df011c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.325863 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.325818 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993f0b44-a09e-44de-b098-d1c973c51d1b-kube-api-access-qn7lc" pod "993f0b44-a09e-44de-b098-d1c973c51d1b" (UID: "993f0b44-a09e-44de-b098-d1c973c51d1b"). InnerVolumeSpecName "kube-api-access-qn7lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:10.326662 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.326640 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-podinfo" pod "993f0b44-a09e-44de-b098-d1c973c51d1b" (UID: "993f0b44-a09e-44de-b098-d1c973c51d1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:24:10.424868 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.424782 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.424868 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.424821 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.424868 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.424833 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn7lc\" (UniqueName: \"kubernetes.io/projected/993f0b44-a09e-44de-b098-d1c973c51d1b-kube-api-access-qn7lc\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.424868 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.424848 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.424868 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.424857 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.424868 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.424866 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.424868 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.424876 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/993f0b44-a09e-44de-b098-d1c973c51d1b-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.425253 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.424885 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd64d741-882d-4727-a3b3-cb9c09df011c-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.425253 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.424894 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6hxx\" (UniqueName: \"kubernetes.io/projected/dd64d741-882d-4727-a3b3-cb9c09df011c-kube-api-access-b6hxx\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.425253 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.424902 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.425253 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.424910 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd64d741-882d-4727-a3b3-cb9c09df011c-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.485020 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.484985 4918 generic.go:356] "Generic (PLEG): container finished" podID="993f0b44-a09e-44de-b098-d1c973c51d1b" containerID="3a46c40647598bf40de4d2cd35856e48f50564768d13f5a498b892d0ab6a14a7" exitCode=0 Nov 03 20:24:10.485246 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.485029 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" Nov 03 20:24:10.485246 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.485065 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" event={"ID":"993f0b44-a09e-44de-b098-d1c973c51d1b","Type":"ContainerDied","Data":"3a46c40647598bf40de4d2cd35856e48f50564768d13f5a498b892d0ab6a14a7"} Nov 03 20:24:10.485246 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.485133 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp" event={"ID":"993f0b44-a09e-44de-b098-d1c973c51d1b","Type":"ContainerDied","Data":"adb99f08d8a8c160aebfda9e1d770e085b2c2b3c81dc538af30cf1093a310c92"} Nov 03 20:24:10.485246 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.485164 4918 scope.go:122] "RemoveContainer" containerID="3a46c40647598bf40de4d2cd35856e48f50564768d13f5a498b892d0ab6a14a7" Nov 03 20:24:10.486678 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.486654 4918 generic.go:356] "Generic (PLEG): container finished" podID="dd64d741-882d-4727-a3b3-cb9c09df011c" containerID="2fe2d628c4b6ad5720b2c7ab434b8a423e98531036306433409cc182ce702542" exitCode=0 Nov 03 20:24:10.486792 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.486687 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" event={"ID":"dd64d741-882d-4727-a3b3-cb9c09df011c","Type":"ContainerDied","Data":"2fe2d628c4b6ad5720b2c7ab434b8a423e98531036306433409cc182ce702542"} Nov 03 20:24:10.486792 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.486713 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" Nov 03 20:24:10.486792 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.486743 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv" event={"ID":"dd64d741-882d-4727-a3b3-cb9c09df011c","Type":"ContainerDied","Data":"4580983d3e33f47e7c1d9bd1832684496e2f87b02693c9ba6ec8d3d3670f0aab"} Nov 03 20:24:10.499617 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.499595 4918 scope.go:122] "RemoveContainer" containerID="3a46c40647598bf40de4d2cd35856e48f50564768d13f5a498b892d0ab6a14a7" Nov 03 20:24:10.499921 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:24:10.499901 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a46c40647598bf40de4d2cd35856e48f50564768d13f5a498b892d0ab6a14a7\": container with ID starting with 3a46c40647598bf40de4d2cd35856e48f50564768d13f5a498b892d0ab6a14a7 not found: ID does not exist" containerID="3a46c40647598bf40de4d2cd35856e48f50564768d13f5a498b892d0ab6a14a7" Nov 03 20:24:10.500000 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.499934 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a46c40647598bf40de4d2cd35856e48f50564768d13f5a498b892d0ab6a14a7"} err="failed to get container status \"3a46c40647598bf40de4d2cd35856e48f50564768d13f5a498b892d0ab6a14a7\": rpc error: code = NotFound desc = could not find container \"3a46c40647598bf40de4d2cd35856e48f50564768d13f5a498b892d0ab6a14a7\": container with ID starting with 3a46c40647598bf40de4d2cd35856e48f50564768d13f5a498b892d0ab6a14a7 not found: ID does not exist" Nov 03 20:24:10.500000 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.499957 4918 scope.go:122] "RemoveContainer" containerID="2fe2d628c4b6ad5720b2c7ab434b8a423e98531036306433409cc182ce702542" Nov 03 20:24:10.513888 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.513862 4918 scope.go:122] "RemoveContainer" containerID="2fe2d628c4b6ad5720b2c7ab434b8a423e98531036306433409cc182ce702542" Nov 03 20:24:10.518833 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:24:10.518791 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fe2d628c4b6ad5720b2c7ab434b8a423e98531036306433409cc182ce702542\": container with ID starting with 2fe2d628c4b6ad5720b2c7ab434b8a423e98531036306433409cc182ce702542 not found: ID does not exist" containerID="2fe2d628c4b6ad5720b2c7ab434b8a423e98531036306433409cc182ce702542" Nov 03 20:24:10.518959 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.518848 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe2d628c4b6ad5720b2c7ab434b8a423e98531036306433409cc182ce702542"} err="failed to get container status \"2fe2d628c4b6ad5720b2c7ab434b8a423e98531036306433409cc182ce702542\": rpc error: code = NotFound desc = could not find container \"2fe2d628c4b6ad5720b2c7ab434b8a423e98531036306433409cc182ce702542\": container with ID starting with 2fe2d628c4b6ad5720b2c7ab434b8a423e98531036306433409cc182ce702542 not found: ID does not exist" Nov 03 20:24:10.533263 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.533172 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp"] Nov 03 20:24:10.538444 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.538415 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-57mgp"] Nov 03 20:24:10.543585 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.543556 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv"] Nov 03 20:24:10.550846 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:10.550820 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-rwxbv"] Nov 03 20:24:11.219237 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:11.219154 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="993f0b44-a09e-44de-b098-d1c973c51d1b" path="/var/lib/kubelet/pods/993f0b44-a09e-44de-b098-d1c973c51d1b/volumes" Nov 03 20:24:11.219782 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:11.219765 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd64d741-882d-4727-a3b3-cb9c09df011c" path="/var/lib/kubelet/pods/dd64d741-882d-4727-a3b3-cb9c09df011c/volumes" Nov 03 20:24:12.012775 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.012747 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:12.120508 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.120409 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/148d938c-d92c-454a-833d-4d2b9b69d9ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148d938c-d92c-454a-833d-4d2b9b69d9ff-istiod-ca-cert\") pod \"148d938c-d92c-454a-833d-4d2b9b69d9ff\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " Nov 03 20:24:12.120508 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.120468 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-token\" (UniqueName: \"kubernetes.io/projected/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-token\") pod \"148d938c-d92c-454a-833d-4d2b9b69d9ff\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " Nov 03 20:24:12.120752 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.120578 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-data\") pod \"148d938c-d92c-454a-833d-4d2b9b69d9ff\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " Nov 03 20:24:12.120752 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.120621 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-podinfo\") pod \"148d938c-d92c-454a-833d-4d2b9b69d9ff\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " Nov 03 20:24:12.120752 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.120652 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-workload-socket\") pod \"148d938c-d92c-454a-833d-4d2b9b69d9ff\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " Nov 03 20:24:12.120752 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.120738 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-credential-socket\") pod \"148d938c-d92c-454a-833d-4d2b9b69d9ff\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " Nov 03 20:24:12.120955 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.120887 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/148d938c-d92c-454a-833d-4d2b9b69d9ff-kube-api-access-jkk9m\" (UniqueName: \"kubernetes.io/projected/148d938c-d92c-454a-833d-4d2b9b69d9ff-kube-api-access-jkk9m\") pod \"148d938c-d92c-454a-833d-4d2b9b69d9ff\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " Nov 03 20:24:12.120955 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.120948 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-workload-certs\") pod \"148d938c-d92c-454a-833d-4d2b9b69d9ff\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " Nov 03 20:24:12.121093 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.121077 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-envoy\") pod \"148d938c-d92c-454a-833d-4d2b9b69d9ff\" (UID: \"148d938c-d92c-454a-833d-4d2b9b69d9ff\") " Nov 03 20:24:12.123478 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.121736 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-credential-socket" pod "148d938c-d92c-454a-833d-4d2b9b69d9ff" (UID: "148d938c-d92c-454a-833d-4d2b9b69d9ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:12.123478 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.121989 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-workload-socket" pod "148d938c-d92c-454a-833d-4d2b9b69d9ff" (UID: "148d938c-d92c-454a-833d-4d2b9b69d9ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:12.123478 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.122641 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:12.123478 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.122665 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:12.123478 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.123306 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/148d938c-d92c-454a-833d-4d2b9b69d9ff-istiod-ca-cert" pod "148d938c-d92c-454a-833d-4d2b9b69d9ff" (UID: "148d938c-d92c-454a-833d-4d2b9b69d9ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:12.123478 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.123356 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-workload-certs" pod "148d938c-d92c-454a-833d-4d2b9b69d9ff" (UID: "148d938c-d92c-454a-833d-4d2b9b69d9ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:12.123478 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.123459 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-data" pod "148d938c-d92c-454a-833d-4d2b9b69d9ff" (UID: "148d938c-d92c-454a-833d-4d2b9b69d9ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:12.126278 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.126249 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-token" pod "148d938c-d92c-454a-833d-4d2b9b69d9ff" (UID: "148d938c-d92c-454a-833d-4d2b9b69d9ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:12.126783 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.126744 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148d938c-d92c-454a-833d-4d2b9b69d9ff-kube-api-access-jkk9m" pod "148d938c-d92c-454a-833d-4d2b9b69d9ff" (UID: "148d938c-d92c-454a-833d-4d2b9b69d9ff"). InnerVolumeSpecName "kube-api-access-jkk9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:12.126887 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.126785 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-podinfo" pod "148d938c-d92c-454a-833d-4d2b9b69d9ff" (UID: "148d938c-d92c-454a-833d-4d2b9b69d9ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:24:12.130350 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.130315 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-envoy" pod "148d938c-d92c-454a-833d-4d2b9b69d9ff" (UID: "148d938c-d92c-454a-833d-4d2b9b69d9ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:12.169329 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.169299 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:12.223863 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.223818 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-workload-certs\") pod \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " Nov 03 20:24:12.224037 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.223945 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-kube-api-access-sgrwj\" (UniqueName: \"kubernetes.io/projected/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-kube-api-access-sgrwj\") pod \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " Nov 03 20:24:12.224037 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.224015 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-workload-socket\") pod \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " Nov 03 20:24:12.224190 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.224164 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-envoy\") pod \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " Nov 03 20:24:12.224250 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.224165 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-workload-certs" pod "3df7a75c-bc28-4e1c-989a-85b9fe3143c7" (UID: "3df7a75c-bc28-4e1c-989a-85b9fe3143c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:12.224357 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.224340 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-podinfo\") pod \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " Nov 03 20:24:12.224433 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.224407 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-workload-socket" pod "3df7a75c-bc28-4e1c-989a-85b9fe3143c7" (UID: "3df7a75c-bc28-4e1c-989a-85b9fe3143c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:12.224486 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.224438 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-token\" (UniqueName: \"kubernetes.io/projected/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-token\") pod \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " Nov 03 20:24:12.224550 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.224486 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istiod-ca-cert\") pod \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " Nov 03 20:24:12.224550 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.224525 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-data\") pod \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " Nov 03 20:24:12.224650 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.224597 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-credential-socket\") pod \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\" (UID: \"3df7a75c-bc28-4e1c-989a-85b9fe3143c7\") " Nov 03 20:24:12.224931 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.224900 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-data" pod "3df7a75c-bc28-4e1c-989a-85b9fe3143c7" (UID: "3df7a75c-bc28-4e1c-989a-85b9fe3143c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:12.225150 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.225128 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-credential-socket" pod "3df7a75c-bc28-4e1c-989a-85b9fe3143c7" (UID: "3df7a75c-bc28-4e1c-989a-85b9fe3143c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:12.225342 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.225319 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istiod-ca-cert" pod "3df7a75c-bc28-4e1c-989a-85b9fe3143c7" (UID: "3df7a75c-bc28-4e1c-989a-85b9fe3143c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:12.225579 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.225557 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:12.225652 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.225585 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:12.225652 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.225602 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:12.225652 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.225613 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:12.225652 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.225626 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:12.225652 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.225640 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkk9m\" (UniqueName: \"kubernetes.io/projected/148d938c-d92c-454a-833d-4d2b9b69d9ff-kube-api-access-jkk9m\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:12.225974 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.225654 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:12.225974 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.225670 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148d938c-d92c-454a-833d-4d2b9b69d9ff-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:12.225974 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.225684 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:12.225974 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.225696 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:12.225974 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.225709 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:12.225974 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.225722 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148d938c-d92c-454a-833d-4d2b9b69d9ff-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:12.226837 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.226804 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-podinfo" pod "3df7a75c-bc28-4e1c-989a-85b9fe3143c7" (UID: "3df7a75c-bc28-4e1c-989a-85b9fe3143c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:24:12.226942 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.226822 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-envoy" pod "3df7a75c-bc28-4e1c-989a-85b9fe3143c7" (UID: "3df7a75c-bc28-4e1c-989a-85b9fe3143c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:12.227266 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.227246 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-kube-api-access-sgrwj" pod "3df7a75c-bc28-4e1c-989a-85b9fe3143c7" (UID: "3df7a75c-bc28-4e1c-989a-85b9fe3143c7"). InnerVolumeSpecName "kube-api-access-sgrwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:12.228276 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.228253 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-token" pod "3df7a75c-bc28-4e1c-989a-85b9fe3143c7" (UID: "3df7a75c-bc28-4e1c-989a-85b9fe3143c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:12.327430 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.327386 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:12.327430 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.327420 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgrwj\" (UniqueName: \"kubernetes.io/projected/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-kube-api-access-sgrwj\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:12.327430 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.327433 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:12.327670 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.327446 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df7a75c-bc28-4e1c-989a-85b9fe3143c7-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:12.509481 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.509382 4918 generic.go:356] "Generic (PLEG): container finished" podID="3df7a75c-bc28-4e1c-989a-85b9fe3143c7" containerID="2c2389939731f479f2fa10bfcddc1251bd88addfe681114ff22f8378989e2adb" exitCode=0 Nov 03 20:24:12.509481 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.509434 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" Nov 03 20:24:12.509481 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.509462 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" event={"ID":"3df7a75c-bc28-4e1c-989a-85b9fe3143c7","Type":"ContainerDied","Data":"2c2389939731f479f2fa10bfcddc1251bd88addfe681114ff22f8378989e2adb"} Nov 03 20:24:12.509777 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.509525 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj" event={"ID":"3df7a75c-bc28-4e1c-989a-85b9fe3143c7","Type":"ContainerDied","Data":"a24c5cc1babc51d6feab592261db7800df1df7c2dd3ee9f86ca22c237dd6be96"} Nov 03 20:24:12.509777 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.509555 4918 scope.go:122] "RemoveContainer" containerID="2c2389939731f479f2fa10bfcddc1251bd88addfe681114ff22f8378989e2adb" Nov 03 20:24:12.512228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.511377 4918 generic.go:356] "Generic (PLEG): container finished" podID="148d938c-d92c-454a-833d-4d2b9b69d9ff" containerID="fa376c2a180c62326d72a1a669a62600221259a91e5f1b15b013337e4a4d566b" exitCode=0 Nov 03 20:24:12.512228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.511402 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" event={"ID":"148d938c-d92c-454a-833d-4d2b9b69d9ff","Type":"ContainerDied","Data":"fa376c2a180c62326d72a1a669a62600221259a91e5f1b15b013337e4a4d566b"} Nov 03 20:24:12.512228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.511432 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" Nov 03 20:24:12.512228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.511482 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4" event={"ID":"148d938c-d92c-454a-833d-4d2b9b69d9ff","Type":"ContainerDied","Data":"d88d1bd1750098677d87025cf47b9292591de665da944e925e3fe92dcc5b5e88"} Nov 03 20:24:12.523648 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.523479 4918 scope.go:122] "RemoveContainer" containerID="2c2389939731f479f2fa10bfcddc1251bd88addfe681114ff22f8378989e2adb" Nov 03 20:24:12.523846 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:24:12.523824 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c2389939731f479f2fa10bfcddc1251bd88addfe681114ff22f8378989e2adb\": container with ID starting with 2c2389939731f479f2fa10bfcddc1251bd88addfe681114ff22f8378989e2adb not found: ID does not exist" containerID="2c2389939731f479f2fa10bfcddc1251bd88addfe681114ff22f8378989e2adb" Nov 03 20:24:12.523931 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.523854 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2389939731f479f2fa10bfcddc1251bd88addfe681114ff22f8378989e2adb"} err="failed to get container status \"2c2389939731f479f2fa10bfcddc1251bd88addfe681114ff22f8378989e2adb\": rpc error: code = NotFound desc = could not find container \"2c2389939731f479f2fa10bfcddc1251bd88addfe681114ff22f8378989e2adb\": container with ID starting with 2c2389939731f479f2fa10bfcddc1251bd88addfe681114ff22f8378989e2adb not found: ID does not exist" Nov 03 20:24:12.523931 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.523874 4918 scope.go:122] "RemoveContainer" containerID="fa376c2a180c62326d72a1a669a62600221259a91e5f1b15b013337e4a4d566b" Nov 03 20:24:12.536545 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.536522 4918 scope.go:122] "RemoveContainer" containerID="fa376c2a180c62326d72a1a669a62600221259a91e5f1b15b013337e4a4d566b" Nov 03 20:24:12.539847 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:24:12.539819 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa376c2a180c62326d72a1a669a62600221259a91e5f1b15b013337e4a4d566b\": container with ID starting with fa376c2a180c62326d72a1a669a62600221259a91e5f1b15b013337e4a4d566b not found: ID does not exist" containerID="fa376c2a180c62326d72a1a669a62600221259a91e5f1b15b013337e4a4d566b" Nov 03 20:24:12.539985 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.539854 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa376c2a180c62326d72a1a669a62600221259a91e5f1b15b013337e4a4d566b"} err="failed to get container status \"fa376c2a180c62326d72a1a669a62600221259a91e5f1b15b013337e4a4d566b\": rpc error: code = NotFound desc = could not find container \"fa376c2a180c62326d72a1a669a62600221259a91e5f1b15b013337e4a4d566b\": container with ID starting with fa376c2a180c62326d72a1a669a62600221259a91e5f1b15b013337e4a4d566b not found: ID does not exist" Nov 03 20:24:12.542371 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.542346 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj"] Nov 03 20:24:12.546721 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.546695 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-gmtsj"] Nov 03 20:24:12.557223 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.557193 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4"] Nov 03 20:24:12.569549 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:12.569515 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-hsgn4"] Nov 03 20:24:13.217698 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:13.217665 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="148d938c-d92c-454a-833d-4d2b9b69d9ff" path="/var/lib/kubelet/pods/148d938c-d92c-454a-833d-4d2b9b69d9ff/volumes" Nov 03 20:24:13.218309 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:13.218262 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3df7a75c-bc28-4e1c-989a-85b9fe3143c7" path="/var/lib/kubelet/pods/3df7a75c-bc28-4e1c-989a-85b9fe3143c7/volumes" Nov 03 20:24:15.482613 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.482577 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc"] Nov 03 20:24:15.483372 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.483355 4918 cpu_manager.go:414] "containerMap: removing container" podUID="148d938c-d92c-454a-833d-4d2b9b69d9ff" containerName="istio-proxy" Nov 03 20:24:15.483422 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.483373 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="148d938c-d92c-454a-833d-4d2b9b69d9ff" containerName="istio-proxy" Nov 03 20:24:15.483457 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.483422 4918 cpu_manager.go:414] "containerMap: removing container" podUID="3df7a75c-bc28-4e1c-989a-85b9fe3143c7" containerName="istio-proxy" Nov 03 20:24:15.483457 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.483427 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3df7a75c-bc28-4e1c-989a-85b9fe3143c7" containerName="istio-proxy" Nov 03 20:24:15.483457 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.483445 4918 cpu_manager.go:414] "containerMap: removing container" podUID="993f0b44-a09e-44de-b098-d1c973c51d1b" containerName="istio-proxy" Nov 03 20:24:15.483457 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.483451 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993f0b44-a09e-44de-b098-d1c973c51d1b" containerName="istio-proxy" Nov 03 20:24:15.483587 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.483459 4918 cpu_manager.go:414] "containerMap: removing container" podUID="dd64d741-882d-4727-a3b3-cb9c09df011c" containerName="istio-proxy" Nov 03 20:24:15.483587 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.483464 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd64d741-882d-4727-a3b3-cb9c09df011c" containerName="istio-proxy" Nov 03 20:24:15.483587 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.483568 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3df7a75c-bc28-4e1c-989a-85b9fe3143c7" containerName="istio-proxy" Nov 03 20:24:15.483587 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.483579 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="993f0b44-a09e-44de-b098-d1c973c51d1b" containerName="istio-proxy" Nov 03 20:24:15.483587 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.483586 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd64d741-882d-4727-a3b3-cb9c09df011c" containerName="istio-proxy" Nov 03 20:24:15.483733 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.483594 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="148d938c-d92c-454a-833d-4d2b9b69d9ff" containerName="istio-proxy" Nov 03 20:24:15.488441 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.488418 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.497840 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.497815 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5ca474-openshift-default-dockercfg-p4nhm\"" Nov 03 20:24:15.506855 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.506832 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc"] Nov 03 20:24:15.581694 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.581657 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2461bee-954b-481d-a740-379482984a4a-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.581863 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.581713 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7qct\" (UniqueName: \"kubernetes.io/projected/b2461bee-954b-481d-a740-379482984a4a-kube-api-access-c7qct\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.581954 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.581932 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.582050 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.582037 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.582087 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.582067 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2461bee-954b-481d-a740-379482984a4a-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.582121 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.582096 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.582155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.582129 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.582228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.582208 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.582301 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.582287 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2461bee-954b-481d-a740-379482984a4a-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.683772 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.683737 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7qct\" (UniqueName: \"kubernetes.io/projected/b2461bee-954b-481d-a740-379482984a4a-kube-api-access-c7qct\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.683942 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.683897 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.684001 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.683977 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.684038 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.684002 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2461bee-954b-481d-a740-379482984a4a-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.684038 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.684027 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.684126 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.684072 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.684126 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.684096 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.684253 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.684196 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2461bee-954b-481d-a740-379482984a4a-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.684301 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.684273 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2461bee-954b-481d-a740-379482984a4a-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.684657 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.684617 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.684657 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.684646 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.684830 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.684671 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.685083 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.685051 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2461bee-954b-481d-a740-379482984a4a-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.685266 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.685246 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.686522 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.686499 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.686687 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.686665 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2461bee-954b-481d-a740-379482984a4a-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.704646 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.704617 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2461bee-954b-481d-a740-379482984a4a-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.710771 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.710739 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7qct\" (UniqueName: \"kubernetes.io/projected/b2461bee-954b-481d-a740-379482984a4a-kube-api-access-c7qct\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:15.810618 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:15.810543 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:16.099238 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:16.099204 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc"] Nov 03 20:24:16.099629 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:24:16.099608 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2461bee_954b_481d_a740_379482984a4a.slice/crio-df14d7c114a19bd4758508752d54fe139f118fa8fa472a48f14b1f6efd9eee4f WatchSource:0}: Error finding container df14d7c114a19bd4758508752d54fe139f118fa8fa472a48f14b1f6efd9eee4f: Status 404 returned error can't find the container with id df14d7c114a19bd4758508752d54fe139f118fa8fa472a48f14b1f6efd9eee4f Nov 03 20:24:16.102513 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:16.102475 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:24:16.102605 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:16.102546 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:24:16.102605 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:16.102578 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:24:16.549073 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:16.548988 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" event={"ID":"b2461bee-954b-481d-a740-379482984a4a","Type":"ContainerStarted","Data":"d63bf761de529013fca363edfa148446ae91f5a1eab8e6c77b46173101569dcf"} Nov 03 20:24:16.549700 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:16.549071 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" event={"ID":"b2461bee-954b-481d-a740-379482984a4a","Type":"ContainerStarted","Data":"df14d7c114a19bd4758508752d54fe139f118fa8fa472a48f14b1f6efd9eee4f"} Nov 03 20:24:16.586592 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:16.586523 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" podStartSLOduration=1.586502458 podStartE2EDuration="1.586502458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:16.57841068 +0000 UTC m=+1639.510278192" watchObservedRunningTime="2026-11-03 20:24:16.586502458 +0000 UTC m=+1639.518369970" Nov 03 20:24:16.811624 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:16.811585 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:17.006139 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.006104 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp"] Nov 03 20:24:17.011745 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.011724 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp" Nov 03 20:24:17.017493 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.017464 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8c684d\"" Nov 03 20:24:17.025264 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.024206 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4"] Nov 03 20:24:17.029566 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.029539 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4" Nov 03 20:24:17.031714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.031604 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp"] Nov 03 20:24:17.032300 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.032020 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8c684d-dockercfg-bs9q8\"" Nov 03 20:24:17.043561 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.040275 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4"] Nov 03 20:24:17.121909 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.121829 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7103c18e-609f-41ed-862a-13a571202b51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp\" (UID: \"7103c18e-609f-41ed-862a-13a571202b51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp" Nov 03 20:24:17.122054 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.122016 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gknq8\" (UniqueName: \"kubernetes.io/projected/7103c18e-609f-41ed-862a-13a571202b51-kube-api-access-gknq8\") pod \"payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp\" (UID: \"7103c18e-609f-41ed-862a-13a571202b51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp" Nov 03 20:24:17.240783 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.240743 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phl2r\" (UniqueName: \"kubernetes.io/projected/3ded5532-c475-49a9-8e49-d4f547837159-kube-api-access-phl2r\") pod \"payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4\" (UID: \"3ded5532-c475-49a9-8e49-d4f547837159\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4" Nov 03 20:24:17.240976 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.240892 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7103c18e-609f-41ed-862a-13a571202b51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp\" (UID: \"7103c18e-609f-41ed-862a-13a571202b51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp" Nov 03 20:24:17.241054 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.241009 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gknq8\" (UniqueName: \"kubernetes.io/projected/7103c18e-609f-41ed-862a-13a571202b51-kube-api-access-gknq8\") pod \"payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp\" (UID: \"7103c18e-609f-41ed-862a-13a571202b51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp" Nov 03 20:24:17.241115 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.241051 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ded5532-c475-49a9-8e49-d4f547837159-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4\" (UID: \"3ded5532-c475-49a9-8e49-d4f547837159\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4" Nov 03 20:24:17.253286 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.242156 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7103c18e-609f-41ed-862a-13a571202b51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp\" (UID: \"7103c18e-609f-41ed-862a-13a571202b51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp" Nov 03 20:24:17.269354 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.269320 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gknq8\" (UniqueName: \"kubernetes.io/projected/7103c18e-609f-41ed-862a-13a571202b51-kube-api-access-gknq8\") pod \"payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp\" (UID: \"7103c18e-609f-41ed-862a-13a571202b51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp" Nov 03 20:24:17.293938 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.293911 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4"] Nov 03 20:24:17.299785 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.299762 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4"] Nov 03 20:24:17.299915 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.299843 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:17.306405 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.306382 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-77e83b\"/\"default-dockercfg-m8f7h\"" Nov 03 20:24:17.306555 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.306507 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-77e83b\"/\"test-model-77e83b-kserve-self-signed-certs\"" Nov 03 20:24:17.307352 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.307329 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-77e83b\"/\"openshift-service-ca.crt\"" Nov 03 20:24:17.307807 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.307626 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-77e83b\"/\"kube-root-ca.crt\"" Nov 03 20:24:17.350214 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.350103 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-model-cache\") pod \"test-model-77e83b-kserve-78ddf686b7-wr9b4\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:17.350432 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.350312 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df7b9bb4-44fb-4272-a6c7-90e71d051f46-tls-certs\") pod \"test-model-77e83b-kserve-78ddf686b7-wr9b4\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:17.350513 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.350431 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp" Nov 03 20:24:17.350513 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.350488 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-home\") pod \"test-model-77e83b-kserve-78ddf686b7-wr9b4\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:17.350635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.350513 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-dshm\") pod \"test-model-77e83b-kserve-78ddf686b7-wr9b4\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:17.350696 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.350644 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ded5532-c475-49a9-8e49-d4f547837159-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4\" (UID: \"3ded5532-c475-49a9-8e49-d4f547837159\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4" Nov 03 20:24:17.350753 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.350699 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdvbr\" (UniqueName: \"kubernetes.io/projected/df7b9bb4-44fb-4272-a6c7-90e71d051f46-kube-api-access-bdvbr\") pod \"test-model-77e83b-kserve-78ddf686b7-wr9b4\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:17.350753 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.350738 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phl2r\" (UniqueName: \"kubernetes.io/projected/3ded5532-c475-49a9-8e49-d4f547837159-kube-api-access-phl2r\") pod \"payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4\" (UID: \"3ded5532-c475-49a9-8e49-d4f547837159\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4" Nov 03 20:24:17.350868 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.350808 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-tmp-dir\") pod \"test-model-77e83b-kserve-78ddf686b7-wr9b4\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:17.351948 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.351914 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ded5532-c475-49a9-8e49-d4f547837159-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4\" (UID: \"3ded5532-c475-49a9-8e49-d4f547837159\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4" Nov 03 20:24:17.364037 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.364009 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phl2r\" (UniqueName: \"kubernetes.io/projected/3ded5532-c475-49a9-8e49-d4f547837159-kube-api-access-phl2r\") pod \"payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4\" (UID: \"3ded5532-c475-49a9-8e49-d4f547837159\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4" Nov 03 20:24:17.466977 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.461615 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdvbr\" (UniqueName: \"kubernetes.io/projected/df7b9bb4-44fb-4272-a6c7-90e71d051f46-kube-api-access-bdvbr\") pod \"test-model-77e83b-kserve-78ddf686b7-wr9b4\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:17.505953 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.489336 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-tmp-dir\") pod \"test-model-77e83b-kserve-78ddf686b7-wr9b4\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:17.505953 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.489420 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-model-cache\") pod \"test-model-77e83b-kserve-78ddf686b7-wr9b4\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:17.505953 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.489568 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df7b9bb4-44fb-4272-a6c7-90e71d051f46-tls-certs\") pod \"test-model-77e83b-kserve-78ddf686b7-wr9b4\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:17.505953 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.489784 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-home\") pod \"test-model-77e83b-kserve-78ddf686b7-wr9b4\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:17.505953 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.489814 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-dshm\") pod \"test-model-77e83b-kserve-78ddf686b7-wr9b4\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:17.505953 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.503819 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-tmp-dir\") pod \"test-model-77e83b-kserve-78ddf686b7-wr9b4\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:17.523601 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.506625 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-home\") pod \"test-model-77e83b-kserve-78ddf686b7-wr9b4\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:17.556230 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.535466 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-dshm\") pod \"test-model-77e83b-kserve-78ddf686b7-wr9b4\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:17.556230 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.536275 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-model-cache\") pod \"test-model-77e83b-kserve-78ddf686b7-wr9b4\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:17.582203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.577918 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df7b9bb4-44fb-4272-a6c7-90e71d051f46-tls-certs\") pod \"test-model-77e83b-kserve-78ddf686b7-wr9b4\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:17.662618 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.661265 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4" Nov 03 20:24:17.818437 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.818209 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdvbr\" (UniqueName: \"kubernetes.io/projected/df7b9bb4-44fb-4272-a6c7-90e71d051f46-kube-api-access-bdvbr\") pod \"test-model-77e83b-kserve-78ddf686b7-wr9b4\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:17.822943 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.822274 4918 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 20:24:17.822943 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.822329 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" podUID="b2461bee-954b-481d-a740-379482984a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 20:24:17.940507 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.940470 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:17.986711 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:17.986682 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp"] Nov 03 20:24:17.994563 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:24:17.994537 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7103c18e_609f_41ed_862a_13a571202b51.slice/crio-ad4839d0c4654601252b935fd886441067c3af98fa9af1b00714350ef6a2a17d WatchSource:0}: Error finding container ad4839d0c4654601252b935fd886441067c3af98fa9af1b00714350ef6a2a17d: Status 404 returned error can't find the container with id ad4839d0c4654601252b935fd886441067c3af98fa9af1b00714350ef6a2a17d Nov 03 20:24:18.179633 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:18.179604 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4"] Nov 03 20:24:18.180237 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:24:18.180204 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ded5532_c475_49a9_8e49_d4f547837159.slice/crio-27181ea6b0fb1b8c946c232dfa53c9349d5b2f4b79b157df6e2289a326e09e65 WatchSource:0}: Error finding container 27181ea6b0fb1b8c946c232dfa53c9349d5b2f4b79b157df6e2289a326e09e65: Status 404 returned error can't find the container with id 27181ea6b0fb1b8c946c232dfa53c9349d5b2f4b79b157df6e2289a326e09e65 Nov 03 20:24:18.357555 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:18.357524 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4"] Nov 03 20:24:18.661439 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:18.660894 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" event={"ID":"df7b9bb4-44fb-4272-a6c7-90e71d051f46","Type":"ContainerStarted","Data":"2ca781493b1a6fb857f62b1617c5270af94dbd7d42a6a7c12bcf80b8d15941be"} Nov 03 20:24:18.661439 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:18.660963 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" event={"ID":"df7b9bb4-44fb-4272-a6c7-90e71d051f46","Type":"ContainerStarted","Data":"57e892041478ce21ef9e0bc0cfb4938aae3d753332997c754e9d76b1ac024a44"} Nov 03 20:24:18.663612 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:18.662955 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp" event={"ID":"7103c18e-609f-41ed-862a-13a571202b51","Type":"ContainerStarted","Data":"f527d1f78e11b17d0d71188cd8c106993be24d7cc59f06a444605b031ef0babd"} Nov 03 20:24:18.663612 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:18.663008 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp" event={"ID":"7103c18e-609f-41ed-862a-13a571202b51","Type":"ContainerStarted","Data":"ad4839d0c4654601252b935fd886441067c3af98fa9af1b00714350ef6a2a17d"} Nov 03 20:24:18.665767 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:18.665379 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp" Nov 03 20:24:18.667361 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:18.667265 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4" event={"ID":"3ded5532-c475-49a9-8e49-d4f547837159","Type":"ContainerStarted","Data":"c8d2d14f271483b1896b4c159a76ca419269f25f8e36e756a3c452d975dbfcdd"} Nov 03 20:24:18.667361 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:18.667307 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4" event={"ID":"3ded5532-c475-49a9-8e49-d4f547837159","Type":"ContainerStarted","Data":"27181ea6b0fb1b8c946c232dfa53c9349d5b2f4b79b157df6e2289a326e09e65"} Nov 03 20:24:18.712901 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:18.712849 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" podStartSLOduration=1.712833003 podStartE2EDuration="1.712833003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:18.711972112 +0000 UTC m=+1641.643839623" watchObservedRunningTime="2026-11-03 20:24:18.712833003 +0000 UTC m=+1641.644700515" Nov 03 20:24:18.749864 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:18.749806 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4" podStartSLOduration=2.749792555 podStartE2EDuration="2.749792555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:18.74711804 +0000 UTC m=+1641.678985584" watchObservedRunningTime="2026-11-03 20:24:18.749792555 +0000 UTC m=+1641.681660063" Nov 03 20:24:18.811650 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:18.811627 4918 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 20:24:18.811806 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:18.811679 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" podUID="b2461bee-954b-481d-a740-379482984a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 20:24:18.941479 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:18.941430 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:19.675918 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:19.675885 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4" Nov 03 20:24:19.811842 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:19.811810 4918 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 20:24:19.811995 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:19.811857 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" podUID="b2461bee-954b-481d-a740-379482984a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 20:24:20.811673 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:20.811636 4918 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 20:24:20.812054 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:20.811691 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" podUID="b2461bee-954b-481d-a740-379482984a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 20:24:21.811544 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:21.811510 4918 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 20:24:21.811727 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:21.811563 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" podUID="b2461bee-954b-481d-a740-379482984a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 20:24:22.810967 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:22.810934 4918 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 20:24:22.811139 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:22.810984 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" podUID="b2461bee-954b-481d-a740-379482984a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 20:24:23.816506 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:23.816472 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:23.816996 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:23.816977 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:23.817780 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:23.817760 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:24:23.841768 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:23.841713 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp" podStartSLOduration=7.841697822 podStartE2EDuration="7.841697822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:18.766355091 +0000 UTC m=+1641.698222602" watchObservedRunningTime="2026-11-03 20:24:23.841697822 +0000 UTC m=+1646.773565333" Nov 03 20:24:27.962449 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:27.962416 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:27.975268 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:27.975238 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:28.212850 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:24:28.212759 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:24:28.771083 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:28.771055 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:24:30.686461 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:30.686425 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp" Nov 03 20:24:30.686870 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:30.686514 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4" Nov 03 20:24:34.934313 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:34.934272 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8bvn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:24:34.935040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:34.935005 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" podUID="ea819fe0-461b-4952-a40e-7a47e8f51580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:39.600760 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.600674 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld"] Nov 03 20:24:39.608000 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.607973 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:39.615464 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.615433 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld"] Nov 03 20:24:39.615832 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.615810 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5de824\"/\"test-model-5de824-kserve-self-signed-certs\"" Nov 03 20:24:39.616006 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.615993 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5de824\"/\"openshift-service-ca.crt\"" Nov 03 20:24:39.616812 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.616792 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5de824\"/\"kube-root-ca.crt\"" Nov 03 20:24:39.617141 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.617122 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5de824\"/\"default-dockercfg-6gdd6\"" Nov 03 20:24:39.728643 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.728607 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t77sp\" (UniqueName: \"kubernetes.io/projected/d852a17f-5c32-4496-b5d7-4a5080e6c33c-kube-api-access-t77sp\") pod \"test-model-5de824-kserve-78c7f5bbd7-xpdld\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:39.728808 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.728683 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d852a17f-5c32-4496-b5d7-4a5080e6c33c-tls-certs\") pod \"test-model-5de824-kserve-78c7f5bbd7-xpdld\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:39.728808 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.728771 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-home\") pod \"test-model-5de824-kserve-78c7f5bbd7-xpdld\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:39.728808 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.728804 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-tmp-dir\") pod \"test-model-5de824-kserve-78c7f5bbd7-xpdld\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:39.728925 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.728824 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-model-cache\") pod \"test-model-5de824-kserve-78c7f5bbd7-xpdld\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:39.728983 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.728964 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-dshm\") pod \"test-model-5de824-kserve-78c7f5bbd7-xpdld\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:39.836199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.833231 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t77sp\" (UniqueName: \"kubernetes.io/projected/d852a17f-5c32-4496-b5d7-4a5080e6c33c-kube-api-access-t77sp\") pod \"test-model-5de824-kserve-78c7f5bbd7-xpdld\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:39.836199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.833335 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d852a17f-5c32-4496-b5d7-4a5080e6c33c-tls-certs\") pod \"test-model-5de824-kserve-78c7f5bbd7-xpdld\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:39.836199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.833404 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-home\") pod \"test-model-5de824-kserve-78c7f5bbd7-xpdld\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:39.836199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.833435 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-tmp-dir\") pod \"test-model-5de824-kserve-78c7f5bbd7-xpdld\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:39.836199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.833463 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-model-cache\") pod \"test-model-5de824-kserve-78c7f5bbd7-xpdld\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:39.836199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.833586 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-dshm\") pod \"test-model-5de824-kserve-78c7f5bbd7-xpdld\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:39.836199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.834608 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-home\") pod \"test-model-5de824-kserve-78c7f5bbd7-xpdld\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:39.836669 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.836519 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-tmp-dir\") pod \"test-model-5de824-kserve-78c7f5bbd7-xpdld\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:39.838520 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.836791 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-model-cache\") pod \"test-model-5de824-kserve-78c7f5bbd7-xpdld\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:39.846195 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.841798 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d852a17f-5c32-4496-b5d7-4a5080e6c33c-tls-certs\") pod \"test-model-5de824-kserve-78c7f5bbd7-xpdld\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:39.846786 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.846754 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-dshm\") pod \"test-model-5de824-kserve-78c7f5bbd7-xpdld\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:39.869252 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.865453 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t77sp\" (UniqueName: \"kubernetes.io/projected/d852a17f-5c32-4496-b5d7-4a5080e6c33c-kube-api-access-t77sp\") pod \"test-model-5de824-kserve-78c7f5bbd7-xpdld\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:39.931520 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:39.931482 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:40.273934 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:24:40.273904 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd852a17f_5c32_4496_b5d7_4a5080e6c33c.slice/crio-106a16f3c1d00590d22af83f71ba19d74b2e3d1e8fa639e4fe4bab846b6f9558 WatchSource:0}: Error finding container 106a16f3c1d00590d22af83f71ba19d74b2e3d1e8fa639e4fe4bab846b6f9558: Status 404 returned error can't find the container with id 106a16f3c1d00590d22af83f71ba19d74b2e3d1e8fa639e4fe4bab846b6f9558 Nov 03 20:24:40.274430 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:40.274404 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld"] Nov 03 20:24:40.989708 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:40.989667 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" event={"ID":"d852a17f-5c32-4496-b5d7-4a5080e6c33c","Type":"ContainerStarted","Data":"8079317e89973ae7894c616027f5147bfbed7aaa5b47ebe923d13bf75c74e3b1"} Nov 03 20:24:40.990108 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:40.989720 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" event={"ID":"d852a17f-5c32-4496-b5d7-4a5080e6c33c","Type":"ContainerStarted","Data":"106a16f3c1d00590d22af83f71ba19d74b2e3d1e8fa639e4fe4bab846b6f9558"} Nov 03 20:24:41.012969 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:41.012918 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" podStartSLOduration=2.012903304 podStartE2EDuration="2.012903304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:41.009392128 +0000 UTC m=+1663.941259638" watchObservedRunningTime="2026-11-03 20:24:41.012903304 +0000 UTC m=+1663.944770815" Nov 03 20:24:41.931772 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:41.931741 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:49.932257 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:49.932217 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:49.947736 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:49.947703 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:50.086202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:50.086137 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:24:51.308251 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.308215 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv"] Nov 03 20:24:51.314666 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.314638 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:51.320900 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.320854 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d9f082\"/\"default-dockercfg-4jd2w\"" Nov 03 20:24:51.324633 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.324606 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv"] Nov 03 20:24:51.324953 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.324927 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d9f082\"/\"kube-root-ca.crt\"" Nov 03 20:24:51.325131 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.325038 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d9f082\"/\"test-model-d9f082-kserve-self-signed-certs\"" Nov 03 20:24:51.325239 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.325149 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d9f082\"/\"openshift-service-ca.crt\"" Nov 03 20:24:51.471421 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.471386 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-home\") pod \"test-model-d9f082-kserve-57687df945-6l8wv\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:51.471623 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.471439 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9bwh\" (UniqueName: \"kubernetes.io/projected/a7765d31-50d2-4227-a7a2-dbb897077456-kube-api-access-j9bwh\") pod \"test-model-d9f082-kserve-57687df945-6l8wv\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:51.471623 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.471545 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7765d31-50d2-4227-a7a2-dbb897077456-tls-certs\") pod \"test-model-d9f082-kserve-57687df945-6l8wv\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:51.471803 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.471785 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-model-cache\") pod \"test-model-d9f082-kserve-57687df945-6l8wv\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:51.471869 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.471854 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-dshm\") pod \"test-model-d9f082-kserve-57687df945-6l8wv\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:51.471924 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.471911 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-tmp-dir\") pod \"test-model-d9f082-kserve-57687df945-6l8wv\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:51.575317 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.574162 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-model-cache\") pod \"test-model-d9f082-kserve-57687df945-6l8wv\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:51.575317 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.574323 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-dshm\") pod \"test-model-d9f082-kserve-57687df945-6l8wv\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:51.575317 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.574419 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-tmp-dir\") pod \"test-model-d9f082-kserve-57687df945-6l8wv\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:51.575317 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.574561 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-home\") pod \"test-model-d9f082-kserve-57687df945-6l8wv\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:51.575317 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.574603 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9bwh\" (UniqueName: \"kubernetes.io/projected/a7765d31-50d2-4227-a7a2-dbb897077456-kube-api-access-j9bwh\") pod \"test-model-d9f082-kserve-57687df945-6l8wv\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:51.575317 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.574649 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7765d31-50d2-4227-a7a2-dbb897077456-tls-certs\") pod \"test-model-d9f082-kserve-57687df945-6l8wv\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:51.581332 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.580661 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-model-cache\") pod \"test-model-d9f082-kserve-57687df945-6l8wv\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:51.581332 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.581083 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-home\") pod \"test-model-d9f082-kserve-57687df945-6l8wv\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:51.582086 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.582024 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-tmp-dir\") pod \"test-model-d9f082-kserve-57687df945-6l8wv\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:51.590715 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.586662 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7765d31-50d2-4227-a7a2-dbb897077456-tls-certs\") pod \"test-model-d9f082-kserve-57687df945-6l8wv\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:51.590715 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.588856 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-dshm\") pod \"test-model-d9f082-kserve-57687df945-6l8wv\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:51.598341 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.598280 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9bwh\" (UniqueName: \"kubernetes.io/projected/a7765d31-50d2-4227-a7a2-dbb897077456-kube-api-access-j9bwh\") pod \"test-model-d9f082-kserve-57687df945-6l8wv\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:51.638130 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.638083 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:51.932737 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:51.932706 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv"] Nov 03 20:24:51.933291 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:24:51.933266 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7765d31_50d2_4227_a7a2_dbb897077456.slice/crio-340fc15ed77f5905677927c297b8630b9fb8ebd71ee572ad446633f64f046131 WatchSource:0}: Error finding container 340fc15ed77f5905677927c297b8630b9fb8ebd71ee572ad446633f64f046131: Status 404 returned error can't find the container with id 340fc15ed77f5905677927c297b8630b9fb8ebd71ee572ad446633f64f046131 Nov 03 20:24:52.097957 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:52.097913 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" event={"ID":"a7765d31-50d2-4227-a7a2-dbb897077456","Type":"ContainerStarted","Data":"5bddbd9625f4c4c22032a58aa6ad9cfeb33b6bcd7e9ea5976112ccd806ae25ae"} Nov 03 20:24:52.098125 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:52.097968 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" event={"ID":"a7765d31-50d2-4227-a7a2-dbb897077456","Type":"ContainerStarted","Data":"340fc15ed77f5905677927c297b8630b9fb8ebd71ee572ad446633f64f046131"} Nov 03 20:24:52.121917 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:52.121866 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" podStartSLOduration=1.121852215 podStartE2EDuration="1.121852215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:52.118753581 +0000 UTC m=+1675.050621092" watchObservedRunningTime="2026-11-03 20:24:52.121852215 +0000 UTC m=+1675.053719726" Nov 03 20:24:52.638464 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:24:52.638420 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:24:58.210161 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:24:58.210126 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:25:00.577571 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.577533 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd"] Nov 03 20:25:00.584844 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.584818 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.609047 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.601366 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj"] Nov 03 20:25:00.612445 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.609814 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.615018 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.614967 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd"] Nov 03 20:25:00.622567 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.621755 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj"] Nov 03 20:25:00.748635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.741620 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6290d9f-ddd4-40d2-809b-110249e01125-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.748635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.741701 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n22kz\" (UniqueName: \"kubernetes.io/projected/0b995af4-64ca-4566-af7f-abd21c9cdb7f-kube-api-access-n22kz\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.748635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.741764 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.748635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.741813 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.748635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.741855 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.748635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.741897 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.748635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.741920 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.748635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.741959 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6290d9f-ddd4-40d2-809b-110249e01125-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.748635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.742011 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6290d9f-ddd4-40d2-809b-110249e01125-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.748635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.742037 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfvch\" (UniqueName: \"kubernetes.io/projected/b6290d9f-ddd4-40d2-809b-110249e01125-kube-api-access-wfvch\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.748635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.742086 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.748635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.747402 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.748635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.747465 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.748635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.747521 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.748635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.747673 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.748635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.747738 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.748635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.747822 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.748635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.747889 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.849922 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.849839 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.849922 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.849892 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.850166 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.849945 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.850166 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.849979 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.850166 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.850039 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6290d9f-ddd4-40d2-809b-110249e01125-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.850166 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.850074 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n22kz\" (UniqueName: \"kubernetes.io/projected/0b995af4-64ca-4566-af7f-abd21c9cdb7f-kube-api-access-n22kz\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.850166 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.850124 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.850166 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.850162 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.850490 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.850247 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.850490 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.850287 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.850490 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.850309 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.850490 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.850347 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6290d9f-ddd4-40d2-809b-110249e01125-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.850490 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.850393 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6290d9f-ddd4-40d2-809b-110249e01125-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.850490 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.850415 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfvch\" (UniqueName: \"kubernetes.io/projected/b6290d9f-ddd4-40d2-809b-110249e01125-kube-api-access-wfvch\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.850490 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.850465 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.850853 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.850499 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.850853 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.850502 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.850853 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.850549 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.850853 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.850613 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.850853 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.850724 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.850853 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.850773 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.850853 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.850510 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.851256 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.851048 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.851256 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.851107 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.851256 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.851205 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.851424 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.851357 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.851767 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.851717 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.851961 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.851939 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6290d9f-ddd4-40d2-809b-110249e01125-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.853049 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.853020 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.853254 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.853233 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.853527 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.853507 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6290d9f-ddd4-40d2-809b-110249e01125-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.853626 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.853597 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.863345 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.863317 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfvch\" (UniqueName: \"kubernetes.io/projected/b6290d9f-ddd4-40d2-809b-110249e01125-kube-api-access-wfvch\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.863454 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.863432 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6290d9f-ddd4-40d2-809b-110249e01125-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:00.863530 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.863484 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.863754 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.863732 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n22kz\" (UniqueName: \"kubernetes.io/projected/0b995af4-64ca-4566-af7f-abd21c9cdb7f-kube-api-access-n22kz\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.931499 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.931466 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:00.943873 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:00.943768 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:01.263567 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:25:01.263540 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6290d9f_ddd4_40d2_809b_110249e01125.slice/crio-c44472434e556fe5aad83a5d4ce2e2686c3d8747f723642dd85706062b931b3c WatchSource:0}: Error finding container c44472434e556fe5aad83a5d4ce2e2686c3d8747f723642dd85706062b931b3c: Status 404 returned error can't find the container with id c44472434e556fe5aad83a5d4ce2e2686c3d8747f723642dd85706062b931b3c Nov 03 20:25:01.265131 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:01.265104 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj"] Nov 03 20:25:01.266015 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:25:01.265991 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b995af4_64ca_4566_af7f_abd21c9cdb7f.slice/crio-7b8784e31dbfba3c35304573b4b48b73d97e2cb0750c4947cf9291f3057ee35c WatchSource:0}: Error finding container 7b8784e31dbfba3c35304573b4b48b73d97e2cb0750c4947cf9291f3057ee35c: Status 404 returned error can't find the container with id 7b8784e31dbfba3c35304573b4b48b73d97e2cb0750c4947cf9291f3057ee35c Nov 03 20:25:01.267170 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:01.267138 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:25:01.267281 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:01.267226 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:25:01.267354 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:01.267275 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:25:01.269350 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:01.269312 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:25:01.269460 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:01.269390 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:25:01.269460 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:01.269434 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:25:01.270252 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:01.270230 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd"] Nov 03 20:25:01.638485 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:01.638447 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:25:01.656734 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:01.656251 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:25:02.203005 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:02.202935 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" event={"ID":"0b995af4-64ca-4566-af7f-abd21c9cdb7f","Type":"ContainerStarted","Data":"c110b7a20ac1757a19ca21526d3af8ac7f5ad63cd9a1a1ccefef00275f571ac7"} Nov 03 20:25:02.203309 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:02.203069 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" event={"ID":"0b995af4-64ca-4566-af7f-abd21c9cdb7f","Type":"ContainerStarted","Data":"7b8784e31dbfba3c35304573b4b48b73d97e2cb0750c4947cf9291f3057ee35c"} Nov 03 20:25:02.213269 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:02.213227 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" event={"ID":"b6290d9f-ddd4-40d2-809b-110249e01125","Type":"ContainerStarted","Data":"3f90c4195c6cc9050988f91e71616e07936195cfa707d38e8fda88008ad2f111"} Nov 03 20:25:02.213475 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:02.213293 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" event={"ID":"b6290d9f-ddd4-40d2-809b-110249e01125","Type":"ContainerStarted","Data":"c44472434e556fe5aad83a5d4ce2e2686c3d8747f723642dd85706062b931b3c"} Nov 03 20:25:02.225977 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:02.224565 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:25:02.284229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:02.282043 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" podStartSLOduration=2.282021799 podStartE2EDuration="2.282021799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:02.260555527 +0000 UTC m=+1685.192423051" watchObservedRunningTime="2026-11-03 20:25:02.282021799 +0000 UTC m=+1685.213889310" Nov 03 20:25:02.953205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:02.949840 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:02.953205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:02.949984 4918 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 20:25:02.953205 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:02.950032 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" podUID="0b995af4-64ca-4566-af7f-abd21c9cdb7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 20:25:02.961200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:02.959512 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:02.966988 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:02.966953 4918 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 03 20:25:02.967251 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:02.967224 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" podUID="b6290d9f-ddd4-40d2-809b-110249e01125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 03 20:25:03.951200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:03.949108 4918 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 03 20:25:03.951200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:03.949205 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" podUID="b6290d9f-ddd4-40d2-809b-110249e01125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 03 20:25:03.951200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:03.949661 4918 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 20:25:03.951200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:03.949711 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" podUID="0b995af4-64ca-4566-af7f-abd21c9cdb7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 20:25:04.944086 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:04.944048 4918 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 03 20:25:04.975345 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:04.972024 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" podUID="b6290d9f-ddd4-40d2-809b-110249e01125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 03 20:25:04.975345 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:04.945483 4918 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 20:25:04.975345 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:04.972249 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" podUID="0b995af4-64ca-4566-af7f-abd21c9cdb7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 20:25:05.935820 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:05.935782 4918 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 20:25:05.936086 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:05.936057 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" podUID="0b995af4-64ca-4566-af7f-abd21c9cdb7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 20:25:05.948814 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:05.948779 4918 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 03 20:25:05.949398 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:05.949366 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" podUID="b6290d9f-ddd4-40d2-809b-110249e01125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 03 20:25:06.986525 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:06.983918 4918 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 03 20:25:06.987822 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:06.987784 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" podUID="b6290d9f-ddd4-40d2-809b-110249e01125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 03 20:25:06.997241 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:06.989225 4918 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 20:25:06.997241 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:06.989278 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" podUID="0b995af4-64ca-4566-af7f-abd21c9cdb7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 20:25:07.933848 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:07.933375 4918 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 20:25:07.933848 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:07.933458 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" podUID="0b995af4-64ca-4566-af7f-abd21c9cdb7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 20:25:07.947960 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:07.947839 4918 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 03 20:25:07.947960 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:07.947898 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" podUID="b6290d9f-ddd4-40d2-809b-110249e01125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 03 20:25:08.981247 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:08.981206 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:09.002725 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:08.982416 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:09.002957 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:09.002813 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:25:09.009082 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:09.008954 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" podStartSLOduration=9.008937592 podStartE2EDuration="9.008937592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:02.357617023 +0000 UTC m=+1685.289484523" watchObservedRunningTime="2026-11-03 20:25:09.008937592 +0000 UTC m=+1691.940805104" Nov 03 20:25:09.131732 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:09.131696 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:09.132248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:09.132220 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:09.133142 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:09.133095 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:25:18.238216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:18.238141 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc"] Nov 03 20:25:18.255161 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:18.255106 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc" Nov 03 20:25:18.263821 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:18.263780 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc"] Nov 03 20:25:18.276952 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:18.276924 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b0adda\"" Nov 03 20:25:18.421200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:18.420418 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b612e8de-dd1e-47db-a7f8-28eb9e4149f9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc\" (UID: \"b612e8de-dd1e-47db-a7f8-28eb9e4149f9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc" Nov 03 20:25:18.421200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:18.420843 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbmzs\" (UniqueName: \"kubernetes.io/projected/b612e8de-dd1e-47db-a7f8-28eb9e4149f9-kube-api-access-rbmzs\") pod \"payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc\" (UID: \"b612e8de-dd1e-47db-a7f8-28eb9e4149f9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc" Nov 03 20:25:18.540328 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:18.533225 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbmzs\" (UniqueName: \"kubernetes.io/projected/b612e8de-dd1e-47db-a7f8-28eb9e4149f9-kube-api-access-rbmzs\") pod \"payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc\" (UID: \"b612e8de-dd1e-47db-a7f8-28eb9e4149f9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc" Nov 03 20:25:18.540328 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:18.533393 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b612e8de-dd1e-47db-a7f8-28eb9e4149f9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc\" (UID: \"b612e8de-dd1e-47db-a7f8-28eb9e4149f9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc" Nov 03 20:25:18.540328 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:18.534502 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b612e8de-dd1e-47db-a7f8-28eb9e4149f9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc\" (UID: \"b612e8de-dd1e-47db-a7f8-28eb9e4149f9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc" Nov 03 20:25:18.594201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:18.584989 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbmzs\" (UniqueName: \"kubernetes.io/projected/b612e8de-dd1e-47db-a7f8-28eb9e4149f9-kube-api-access-rbmzs\") pod \"payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc\" (UID: \"b612e8de-dd1e-47db-a7f8-28eb9e4149f9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc" Nov 03 20:25:18.594201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:18.589580 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc" Nov 03 20:25:18.984563 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:25:18.983452 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb612e8de_dd1e_47db_a7f8_28eb9e4149f9.slice/crio-1a5fcd1c0bca7f1186e92f52ebde7137d10a0d500e1f9a31bd91c4314618c5ae WatchSource:0}: Error finding container 1a5fcd1c0bca7f1186e92f52ebde7137d10a0d500e1f9a31bd91c4314618c5ae: Status 404 returned error can't find the container with id 1a5fcd1c0bca7f1186e92f52ebde7137d10a0d500e1f9a31bd91c4314618c5ae Nov 03 20:25:18.999453 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:18.999415 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc"] Nov 03 20:25:19.584709 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:19.584672 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc" event={"ID":"b612e8de-dd1e-47db-a7f8-28eb9e4149f9","Type":"ContainerStarted","Data":"c571d830d0a97278b8c9a56fd3e53d62cf83d796e7b90c1532b4e4af80e59904"} Nov 03 20:25:19.585224 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:19.584728 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc" event={"ID":"b612e8de-dd1e-47db-a7f8-28eb9e4149f9","Type":"ContainerStarted","Data":"1a5fcd1c0bca7f1186e92f52ebde7137d10a0d500e1f9a31bd91c4314618c5ae"} Nov 03 20:25:19.585224 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:19.584815 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc" Nov 03 20:25:19.608786 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:19.608732 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc" podStartSLOduration=1.6087155960000001 podStartE2EDuration="1.608715596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:19.602896789 +0000 UTC m=+1702.534764302" watchObservedRunningTime="2026-11-03 20:25:19.608715596 +0000 UTC m=+1702.540583106" Nov 03 20:25:28.214682 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:25:28.214635 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:25:30.596116 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:30.596084 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc" Nov 03 20:25:33.062890 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:33.062840 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr"] Nov 03 20:25:33.063443 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:33.063162 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" podUID="370a7106-3c26-451f-8ad2-76a27a576f64" containerName="istio-proxy" containerID="cri-o://c7a585b18ff8f3f2f738b5e81788d39d50a8f4536a42334b444a01bd333a7165" gracePeriod=30 Nov 03 20:25:33.066789 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:33.066763 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd"] Nov 03 20:25:33.067025 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:33.066974 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" podUID="75fc50e4-2ec7-4e3f-bf44-0572e998c25d" containerName="istio-proxy" containerID="cri-o://b6960ce300cc92c859d71a6a21876ae2befc85c4fc0d0df757521511f8c5c36d" gracePeriod=30 Nov 03 20:25:38.247710 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.247675 4918 generic.go:356] "Generic (PLEG): container finished" podID="75fc50e4-2ec7-4e3f-bf44-0572e998c25d" containerID="b6960ce300cc92c859d71a6a21876ae2befc85c4fc0d0df757521511f8c5c36d" exitCode=0 Nov 03 20:25:38.248117 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.247767 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" event={"ID":"75fc50e4-2ec7-4e3f-bf44-0572e998c25d","Type":"ContainerDied","Data":"b6960ce300cc92c859d71a6a21876ae2befc85c4fc0d0df757521511f8c5c36d"} Nov 03 20:25:38.249578 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.249553 4918 generic.go:356] "Generic (PLEG): container finished" podID="370a7106-3c26-451f-8ad2-76a27a576f64" containerID="c7a585b18ff8f3f2f738b5e81788d39d50a8f4536a42334b444a01bd333a7165" exitCode=0 Nov 03 20:25:38.249703 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.249662 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" event={"ID":"370a7106-3c26-451f-8ad2-76a27a576f64","Type":"ContainerDied","Data":"c7a585b18ff8f3f2f738b5e81788d39d50a8f4536a42334b444a01bd333a7165"} Nov 03 20:25:38.414207 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.414169 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:25:38.461488 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.461428 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/370a7106-3c26-451f-8ad2-76a27a576f64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370a7106-3c26-451f-8ad2-76a27a576f64-istio-podinfo\") pod \"370a7106-3c26-451f-8ad2-76a27a576f64\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " Nov 03 20:25:38.461637 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.461526 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370a7106-3c26-451f-8ad2-76a27a576f64-istio-token\" (UniqueName: \"kubernetes.io/projected/370a7106-3c26-451f-8ad2-76a27a576f64-istio-token\") pod \"370a7106-3c26-451f-8ad2-76a27a576f64\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " Nov 03 20:25:38.461940 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.461917 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-istio-envoy\") pod \"370a7106-3c26-451f-8ad2-76a27a576f64\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " Nov 03 20:25:38.462163 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.462142 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-istio-data\") pod \"370a7106-3c26-451f-8ad2-76a27a576f64\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " Nov 03 20:25:38.462343 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.462322 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-credential-socket\") pod \"370a7106-3c26-451f-8ad2-76a27a576f64\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " Nov 03 20:25:38.462564 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.462543 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-workload-socket\") pod \"370a7106-3c26-451f-8ad2-76a27a576f64\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " Nov 03 20:25:38.462857 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.462832 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370a7106-3c26-451f-8ad2-76a27a576f64-kube-api-access-5dqv6\" (UniqueName: \"kubernetes.io/projected/370a7106-3c26-451f-8ad2-76a27a576f64-kube-api-access-5dqv6\") pod \"370a7106-3c26-451f-8ad2-76a27a576f64\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " Nov 03 20:25:38.463216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.462578 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-credential-socket" pod "370a7106-3c26-451f-8ad2-76a27a576f64" (UID: "370a7106-3c26-451f-8ad2-76a27a576f64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:38.463216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.462929 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-workload-socket" pod "370a7106-3c26-451f-8ad2-76a27a576f64" (UID: "370a7106-3c26-451f-8ad2-76a27a576f64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:38.463496 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.463469 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-workload-certs\") pod \"370a7106-3c26-451f-8ad2-76a27a576f64\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " Nov 03 20:25:38.463688 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.463666 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/370a7106-3c26-451f-8ad2-76a27a576f64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370a7106-3c26-451f-8ad2-76a27a576f64-istiod-ca-cert\") pod \"370a7106-3c26-451f-8ad2-76a27a576f64\" (UID: \"370a7106-3c26-451f-8ad2-76a27a576f64\") " Nov 03 20:25:38.464217 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.464052 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/370a7106-3c26-451f-8ad2-76a27a576f64-istiod-ca-cert" pod "370a7106-3c26-451f-8ad2-76a27a576f64" (UID: "370a7106-3c26-451f-8ad2-76a27a576f64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:38.464614 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.464589 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-workload-certs" pod "370a7106-3c26-451f-8ad2-76a27a576f64" (UID: "370a7106-3c26-451f-8ad2-76a27a576f64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:38.464746 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.464611 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370a7106-3c26-451f-8ad2-76a27a576f64-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.464995 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.464977 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.465111 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.465095 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.465111 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.465019 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/370a7106-3c26-451f-8ad2-76a27a576f64-istio-podinfo" pod "370a7106-3c26-451f-8ad2-76a27a576f64" (UID: "370a7106-3c26-451f-8ad2-76a27a576f64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:25:38.465316 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.465081 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-istio-envoy" pod "370a7106-3c26-451f-8ad2-76a27a576f64" (UID: "370a7106-3c26-451f-8ad2-76a27a576f64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:38.465316 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.465212 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-istio-data" pod "370a7106-3c26-451f-8ad2-76a27a576f64" (UID: "370a7106-3c26-451f-8ad2-76a27a576f64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:38.465835 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.465558 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:25:38.466400 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.466355 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370a7106-3c26-451f-8ad2-76a27a576f64-kube-api-access-5dqv6" pod "370a7106-3c26-451f-8ad2-76a27a576f64" (UID: "370a7106-3c26-451f-8ad2-76a27a576f64"). InnerVolumeSpecName "kube-api-access-5dqv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:38.466748 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.466716 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370a7106-3c26-451f-8ad2-76a27a576f64-istio-token" pod "370a7106-3c26-451f-8ad2-76a27a576f64" (UID: "370a7106-3c26-451f-8ad2-76a27a576f64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:38.566549 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.566500 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-credential-socket\") pod \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " Nov 03 20:25:38.566719 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.566642 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-envoy\") pod \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " Nov 03 20:25:38.566719 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.566681 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istiod-ca-cert\") pod \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " Nov 03 20:25:38.566792 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.566729 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-data\") pod \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " Nov 03 20:25:38.566834 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.566775 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-credential-socket" pod "75fc50e4-2ec7-4e3f-bf44-0572e998c25d" (UID: "75fc50e4-2ec7-4e3f-bf44-0572e998c25d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:38.566873 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.566839 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-kube-api-access-4hzr2\" (UniqueName: \"kubernetes.io/projected/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-kube-api-access-4hzr2\") pod \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " Nov 03 20:25:38.566923 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.566907 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-workload-socket\") pod \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " Nov 03 20:25:38.567083 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.567043 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-data" pod "75fc50e4-2ec7-4e3f-bf44-0572e998c25d" (UID: "75fc50e4-2ec7-4e3f-bf44-0572e998c25d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:38.567083 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.567076 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istiod-ca-cert" pod "75fc50e4-2ec7-4e3f-bf44-0572e998c25d" (UID: "75fc50e4-2ec7-4e3f-bf44-0572e998c25d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:38.567309 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.567154 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-workload-certs\") pod \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " Nov 03 20:25:38.567309 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.567281 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-workload-socket" pod "75fc50e4-2ec7-4e3f-bf44-0572e998c25d" (UID: "75fc50e4-2ec7-4e3f-bf44-0572e998c25d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:38.567426 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.567340 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-podinfo\") pod \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " Nov 03 20:25:38.567426 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.567407 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-workload-certs" pod "75fc50e4-2ec7-4e3f-bf44-0572e998c25d" (UID: "75fc50e4-2ec7-4e3f-bf44-0572e998c25d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:38.567541 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.567516 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-token\" (UniqueName: \"kubernetes.io/projected/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-token\") pod \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\" (UID: \"75fc50e4-2ec7-4e3f-bf44-0572e998c25d\") " Nov 03 20:25:38.568476 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.568449 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dqv6\" (UniqueName: \"kubernetes.io/projected/370a7106-3c26-451f-8ad2-76a27a576f64-kube-api-access-5dqv6\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.568609 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.568483 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.568609 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.568503 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.568609 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.568521 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.568609 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.568536 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.568609 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.568550 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.568609 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.568565 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370a7106-3c26-451f-8ad2-76a27a576f64-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.568609 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.568578 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370a7106-3c26-451f-8ad2-76a27a576f64-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.568609 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.568593 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.568609 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.568607 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370a7106-3c26-451f-8ad2-76a27a576f64-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.568922 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.568621 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.569072 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.569047 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-envoy" pod "75fc50e4-2ec7-4e3f-bf44-0572e998c25d" (UID: "75fc50e4-2ec7-4e3f-bf44-0572e998c25d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:38.569480 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.569461 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-kube-api-access-4hzr2" pod "75fc50e4-2ec7-4e3f-bf44-0572e998c25d" (UID: "75fc50e4-2ec7-4e3f-bf44-0572e998c25d"). InnerVolumeSpecName "kube-api-access-4hzr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:38.569642 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.569614 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-token" pod "75fc50e4-2ec7-4e3f-bf44-0572e998c25d" (UID: "75fc50e4-2ec7-4e3f-bf44-0572e998c25d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:38.569720 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.569653 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-podinfo" pod "75fc50e4-2ec7-4e3f-bf44-0572e998c25d" (UID: "75fc50e4-2ec7-4e3f-bf44-0572e998c25d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:25:38.670786 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.670709 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.670786 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.670739 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.670786 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.670749 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.670786 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:38.670758 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hzr2\" (UniqueName: \"kubernetes.io/projected/75fc50e4-2ec7-4e3f-bf44-0572e998c25d-kube-api-access-4hzr2\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:39.259079 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.259046 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" Nov 03 20:25:39.259079 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.259066 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd" event={"ID":"75fc50e4-2ec7-4e3f-bf44-0572e998c25d","Type":"ContainerDied","Data":"d6b833cf8f0123bac318021995480a161b5730b0950f0875a18d11a6eb22cef6"} Nov 03 20:25:39.259584 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.259131 4918 scope.go:122] "RemoveContainer" containerID="b6960ce300cc92c859d71a6a21876ae2befc85c4fc0d0df757521511f8c5c36d" Nov 03 20:25:39.260754 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.260727 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" Nov 03 20:25:39.260877 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.260761 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr" event={"ID":"370a7106-3c26-451f-8ad2-76a27a576f64","Type":"ContainerDied","Data":"066ad8bc1eb5d89cc84f0131751f933d3cb4b816ea2cf239d8278f7c6c6da114"} Nov 03 20:25:39.270753 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.270734 4918 scope.go:122] "RemoveContainer" containerID="c7a585b18ff8f3f2f738b5e81788d39d50a8f4536a42334b444a01bd333a7165" Nov 03 20:25:39.284238 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.284194 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd"] Nov 03 20:25:39.289260 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.289237 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-t78fd"] Nov 03 20:25:39.303142 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.303118 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr"] Nov 03 20:25:39.307299 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.307276 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-gx8tr"] Nov 03 20:25:39.525762 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.525672 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb"] Nov 03 20:25:39.526569 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.526546 4918 cpu_manager.go:414] "containerMap: removing container" podUID="370a7106-3c26-451f-8ad2-76a27a576f64" containerName="istio-proxy" Nov 03 20:25:39.526569 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.526567 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="370a7106-3c26-451f-8ad2-76a27a576f64" containerName="istio-proxy" Nov 03 20:25:39.526733 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.526600 4918 cpu_manager.go:414] "containerMap: removing container" podUID="75fc50e4-2ec7-4e3f-bf44-0572e998c25d" containerName="istio-proxy" Nov 03 20:25:39.526733 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.526605 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75fc50e4-2ec7-4e3f-bf44-0572e998c25d" containerName="istio-proxy" Nov 03 20:25:39.526733 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.526728 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="75fc50e4-2ec7-4e3f-bf44-0572e998c25d" containerName="istio-proxy" Nov 03 20:25:39.526924 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.526743 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="370a7106-3c26-451f-8ad2-76a27a576f64" containerName="istio-proxy" Nov 03 20:25:39.548907 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.548827 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" Nov 03 20:25:39.557733 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.557683 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b0adda-serving-cert\"" Nov 03 20:25:39.571376 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.571347 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb"] Nov 03 20:25:39.587730 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.587689 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14e66787-45ea-4e62-83f8-e14a0388b915-metrics-certs\") pod \"maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb\" (UID: \"14e66787-45ea-4e62-83f8-e14a0388b915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" Nov 03 20:25:39.587886 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.587745 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e66787-45ea-4e62-83f8-e14a0388b915-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb\" (UID: \"14e66787-45ea-4e62-83f8-e14a0388b915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" Nov 03 20:25:39.588000 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.587969 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14e66787-45ea-4e62-83f8-e14a0388b915-maas-api-tls\") pod \"maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb\" (UID: \"14e66787-45ea-4e62-83f8-e14a0388b915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" Nov 03 20:25:39.588141 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.588121 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt7dd\" (UniqueName: \"kubernetes.io/projected/14e66787-45ea-4e62-83f8-e14a0388b915-kube-api-access-tt7dd\") pod \"maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb\" (UID: \"14e66787-45ea-4e62-83f8-e14a0388b915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" Nov 03 20:25:39.690378 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.690339 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt7dd\" (UniqueName: \"kubernetes.io/projected/14e66787-45ea-4e62-83f8-e14a0388b915-kube-api-access-tt7dd\") pod \"maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb\" (UID: \"14e66787-45ea-4e62-83f8-e14a0388b915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" Nov 03 20:25:39.690605 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.690585 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14e66787-45ea-4e62-83f8-e14a0388b915-metrics-certs\") pod \"maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb\" (UID: \"14e66787-45ea-4e62-83f8-e14a0388b915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" Nov 03 20:25:39.690682 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.690633 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e66787-45ea-4e62-83f8-e14a0388b915-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb\" (UID: \"14e66787-45ea-4e62-83f8-e14a0388b915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" Nov 03 20:25:39.690776 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.690762 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14e66787-45ea-4e62-83f8-e14a0388b915-maas-api-tls\") pod \"maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb\" (UID: \"14e66787-45ea-4e62-83f8-e14a0388b915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" Nov 03 20:25:39.691366 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.691341 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e66787-45ea-4e62-83f8-e14a0388b915-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb\" (UID: \"14e66787-45ea-4e62-83f8-e14a0388b915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" Nov 03 20:25:39.693099 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.693078 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14e66787-45ea-4e62-83f8-e14a0388b915-metrics-certs\") pod \"maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb\" (UID: \"14e66787-45ea-4e62-83f8-e14a0388b915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" Nov 03 20:25:39.693271 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.693219 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14e66787-45ea-4e62-83f8-e14a0388b915-maas-api-tls\") pod \"maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb\" (UID: \"14e66787-45ea-4e62-83f8-e14a0388b915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" Nov 03 20:25:39.702729 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.702708 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt7dd\" (UniqueName: \"kubernetes.io/projected/14e66787-45ea-4e62-83f8-e14a0388b915-kube-api-access-tt7dd\") pod \"maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb\" (UID: \"14e66787-45ea-4e62-83f8-e14a0388b915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" Nov 03 20:25:39.875425 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:39.875386 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" Nov 03 20:25:40.132258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.130942 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc"] Nov 03 20:25:40.160084 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.157869 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.160084 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.160067 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc"] Nov 03 20:25:40.163285 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.163252 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c2c601-openshift-default-dockercfg-zq6pw\"" Nov 03 20:25:40.218285 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.218249 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwxgg\" (UniqueName: \"kubernetes.io/projected/3e465e5f-cb99-475a-848a-a182ceccc56e-kube-api-access-wwxgg\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.220334 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.220302 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-data\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.220663 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.220643 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e465e5f-cb99-475a-848a-a182ceccc56e-istiod-ca-cert\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.221745 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.221723 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-token\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.222260 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.222116 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-envoy\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.222419 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.222400 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-workload-socket\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.222759 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.222740 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-podinfo\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.222981 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.222966 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-credential-socket\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.223739 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.223721 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-workload-certs\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.243157 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.243111 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb"] Nov 03 20:25:40.407754 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.406308 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwxgg\" (UniqueName: \"kubernetes.io/projected/3e465e5f-cb99-475a-848a-a182ceccc56e-kube-api-access-wwxgg\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.407754 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.406407 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-data\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.407754 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.406560 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e465e5f-cb99-475a-848a-a182ceccc56e-istiod-ca-cert\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.407754 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.406588 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-token\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.407754 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.406715 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-envoy\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.407754 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.406745 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-workload-socket\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.407754 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.406777 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-podinfo\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.407754 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.406878 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-credential-socket\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.407754 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.406909 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-workload-certs\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.407754 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.407626 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-workload-certs\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.408853 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.408830 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-data\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.410922 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.409558 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-workload-socket\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.429207 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.414646 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-credential-socket\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.429207 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.415073 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e465e5f-cb99-475a-848a-a182ceccc56e-istiod-ca-cert\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.438943 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.435585 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-envoy\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.438943 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.436090 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-podinfo\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.455039 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.454453 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwxgg\" (UniqueName: \"kubernetes.io/projected/3e465e5f-cb99-475a-848a-a182ceccc56e-kube-api-access-wwxgg\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.455693 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.455619 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-token\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.457437 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.456217 4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:25:40.525404 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.525371 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:40.823103 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:25:40.823072 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e465e5f_cb99_475a_848a_a182ceccc56e.slice/crio-16c1b6ff78457e80a84adb84bdb717265573f71d8c1b8d6fc206acf4ce17c663 WatchSource:0}: Error finding container 16c1b6ff78457e80a84adb84bdb717265573f71d8c1b8d6fc206acf4ce17c663: Status 404 returned error can't find the container with id 16c1b6ff78457e80a84adb84bdb717265573f71d8c1b8d6fc206acf4ce17c663 Nov 03 20:25:40.823666 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.823644 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc"] Nov 03 20:25:40.826055 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.826019 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:25:40.826121 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.826088 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:25:40.826121 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:40.826116 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:25:41.300277 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:41.292299 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="370a7106-3c26-451f-8ad2-76a27a576f64" path="/var/lib/kubelet/pods/370a7106-3c26-451f-8ad2-76a27a576f64/volumes" Nov 03 20:25:41.300277 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:41.293129 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75fc50e4-2ec7-4e3f-bf44-0572e998c25d" path="/var/lib/kubelet/pods/75fc50e4-2ec7-4e3f-bf44-0572e998c25d/volumes" Nov 03 20:25:41.468724 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:41.468682 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" event={"ID":"14e66787-45ea-4e62-83f8-e14a0388b915","Type":"ContainerStarted","Data":"adc37e0e1df0f755ccac79a67244d11d4338893f1a82296b1eaf9b814506a480"} Nov 03 20:25:41.469245 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:41.468752 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" event={"ID":"14e66787-45ea-4e62-83f8-e14a0388b915","Type":"ContainerStarted","Data":"9381c32fc3c3656afe9e62e1168fb72f19bd930968ebbac25e4144c7276b974f"} Nov 03 20:25:41.469245 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:41.468784 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" Nov 03 20:25:41.471245 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:41.470910 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" event={"ID":"3e465e5f-cb99-475a-848a-a182ceccc56e","Type":"ContainerStarted","Data":"b86ad3a64e81e11414b069a145f97d32fe73d2ef95172bb327f773f85d26732c"} Nov 03 20:25:41.471245 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:41.470967 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" event={"ID":"3e465e5f-cb99-475a-848a-a182ceccc56e","Type":"ContainerStarted","Data":"16c1b6ff78457e80a84adb84bdb717265573f71d8c1b8d6fc206acf4ce17c663"} Nov 03 20:25:41.492852 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:41.492794 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" podStartSLOduration=1.994438989 podStartE2EDuration="2.492777321s" totalImagesPullingTime="498.338332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:41.488499621 +0000 UTC m=+1724.420367136" watchObservedRunningTime="2026-11-03 20:25:41.492777321 +0000 UTC m=+1724.424644834" Nov 03 20:25:41.515667 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:41.515618 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" podStartSLOduration=1.515602219 podStartE2EDuration="1.515602219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:41.510939339 +0000 UTC m=+1724.442806850" watchObservedRunningTime="2026-11-03 20:25:41.515602219 +0000 UTC m=+1724.447469730" Nov 03 20:25:41.525726 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:41.525688 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:41.527271 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:41.527240 4918 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 20:25:41.527390 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:41.527300 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" podUID="3e465e5f-cb99-475a-848a-a182ceccc56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 20:25:42.530780 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:42.530749 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:43.487275 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:43.487237 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:43.488142 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:43.488123 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:25:47.485534 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:25:47.485498 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" Nov 03 20:25:58.213050 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:25:58.213001 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:26:14.058739 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.058700 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92"] Nov 03 20:26:14.065143 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.065117 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:14.075705 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.075672 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92"] Nov 03 20:26:14.076950 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.076073 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d0898b\"/\"default-dockercfg-d8h49\"" Nov 03 20:26:14.077114 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.076114 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d0898b\"/\"openshift-service-ca.crt\"" Nov 03 20:26:14.087412 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.076170 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d0898b\"/\"test-model-d0898b-kserve-self-signed-certs\"" Nov 03 20:26:14.087600 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.076265 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d0898b\"/\"kube-root-ca.crt\"" Nov 03 20:26:14.143769 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.143730 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1203aec-cde6-46b4-acf9-991475721fd7-tls-certs\") pod \"test-model-d0898b-kserve-7977b76d69-wtr92\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:14.143946 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.143922 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-home\") pod \"test-model-d0898b-kserve-7977b76d69-wtr92\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:14.144054 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.144037 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-model-cache\") pod \"test-model-d0898b-kserve-7977b76d69-wtr92\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:14.144099 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.144082 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-dshm\") pod \"test-model-d0898b-kserve-7977b76d69-wtr92\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:14.144157 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.144110 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dcbw\" (UniqueName: \"kubernetes.io/projected/b1203aec-cde6-46b4-acf9-991475721fd7-kube-api-access-2dcbw\") pod \"test-model-d0898b-kserve-7977b76d69-wtr92\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:14.144376 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.144350 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-tmp-dir\") pod \"test-model-d0898b-kserve-7977b76d69-wtr92\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:14.286247 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.278153 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-tmp-dir\") pod \"test-model-d0898b-kserve-7977b76d69-wtr92\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:14.286247 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.278299 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1203aec-cde6-46b4-acf9-991475721fd7-tls-certs\") pod \"test-model-d0898b-kserve-7977b76d69-wtr92\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:14.286247 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.278444 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-home\") pod \"test-model-d0898b-kserve-7977b76d69-wtr92\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:14.286247 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.278582 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-model-cache\") pod \"test-model-d0898b-kserve-7977b76d69-wtr92\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:14.286247 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.278639 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-dshm\") pod \"test-model-d0898b-kserve-7977b76d69-wtr92\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:14.286247 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.278671 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dcbw\" (UniqueName: \"kubernetes.io/projected/b1203aec-cde6-46b4-acf9-991475721fd7-kube-api-access-2dcbw\") pod \"test-model-d0898b-kserve-7977b76d69-wtr92\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:14.286247 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.279576 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-tmp-dir\") pod \"test-model-d0898b-kserve-7977b76d69-wtr92\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:14.286247 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.285418 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-model-cache\") pod \"test-model-d0898b-kserve-7977b76d69-wtr92\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:14.286247 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.285639 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-home\") pod \"test-model-d0898b-kserve-7977b76d69-wtr92\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:14.323241 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.322999 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dcbw\" (UniqueName: \"kubernetes.io/projected/b1203aec-cde6-46b4-acf9-991475721fd7-kube-api-access-2dcbw\") pod \"test-model-d0898b-kserve-7977b76d69-wtr92\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:14.362202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.361377 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-dshm\") pod \"test-model-d0898b-kserve-7977b76d69-wtr92\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:14.367207 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.363059 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1203aec-cde6-46b4-acf9-991475721fd7-tls-certs\") pod \"test-model-d0898b-kserve-7977b76d69-wtr92\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:14.415834 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.414538 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:14.742479 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.742447 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92"] Nov 03 20:26:14.742975 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:26:14.742944 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1203aec_cde6_46b4_acf9_991475721fd7.slice/crio-7ae43ead503a93a31cddd389119c61bb476b2bb6024a5d8d38ac5ce06e860be4 WatchSource:0}: Error finding container 7ae43ead503a93a31cddd389119c61bb476b2bb6024a5d8d38ac5ce06e860be4: Status 404 returned error can't find the container with id 7ae43ead503a93a31cddd389119c61bb476b2bb6024a5d8d38ac5ce06e860be4 Nov 03 20:26:14.811517 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:14.811488 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" event={"ID":"b1203aec-cde6-46b4-acf9-991475721fd7","Type":"ContainerStarted","Data":"7ae43ead503a93a31cddd389119c61bb476b2bb6024a5d8d38ac5ce06e860be4"} Nov 03 20:26:15.246888 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.246855 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2"] Nov 03 20:26:15.251844 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.251818 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:15.258088 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.258063 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8c684d\"/\"openshift-service-ca.crt\"" Nov 03 20:26:15.259744 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.259726 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8c684d\"/\"kube-root-ca.crt\"" Nov 03 20:26:15.260203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.260068 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8c684d\"/\"default-dockercfg-df8pd\"" Nov 03 20:26:15.260203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.260157 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8c684d\"/\"test-model-8c684d-kserve-self-signed-certs\"" Nov 03 20:26:15.263586 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.263400 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2"] Nov 03 20:26:15.411062 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.411023 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thf76\" (UniqueName: \"kubernetes.io/projected/a842122e-88c2-4a73-aabf-727008e66cd0-kube-api-access-thf76\") pod \"test-model-8c684d-kserve-6899d979d5-gpqr2\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:15.411250 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.411157 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a842122e-88c2-4a73-aabf-727008e66cd0-tls-certs\") pod \"test-model-8c684d-kserve-6899d979d5-gpqr2\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:15.411250 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.411245 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-tmp-dir\") pod \"test-model-8c684d-kserve-6899d979d5-gpqr2\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:15.411330 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.411305 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-dshm\") pod \"test-model-8c684d-kserve-6899d979d5-gpqr2\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:15.411375 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.411361 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-model-cache\") pod \"test-model-8c684d-kserve-6899d979d5-gpqr2\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:15.411572 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.411554 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-home\") pod \"test-model-8c684d-kserve-6899d979d5-gpqr2\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:15.543275 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.541465 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-home\") pod \"test-model-8c684d-kserve-6899d979d5-gpqr2\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:15.543275 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.541617 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thf76\" (UniqueName: \"kubernetes.io/projected/a842122e-88c2-4a73-aabf-727008e66cd0-kube-api-access-thf76\") pod \"test-model-8c684d-kserve-6899d979d5-gpqr2\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:15.543275 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.541710 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a842122e-88c2-4a73-aabf-727008e66cd0-tls-certs\") pod \"test-model-8c684d-kserve-6899d979d5-gpqr2\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:15.543275 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.541798 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-tmp-dir\") pod \"test-model-8c684d-kserve-6899d979d5-gpqr2\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:15.543275 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.541827 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-dshm\") pod \"test-model-8c684d-kserve-6899d979d5-gpqr2\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:15.543275 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.541889 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-model-cache\") pod \"test-model-8c684d-kserve-6899d979d5-gpqr2\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:15.543275 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.542770 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-home\") pod \"test-model-8c684d-kserve-6899d979d5-gpqr2\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:15.555198 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.551512 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-model-cache\") pod \"test-model-8c684d-kserve-6899d979d5-gpqr2\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:15.555198 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.551747 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-tmp-dir\") pod \"test-model-8c684d-kserve-6899d979d5-gpqr2\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:15.572204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.568745 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-dshm\") pod \"test-model-8c684d-kserve-6899d979d5-gpqr2\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:15.578230 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.572973 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thf76\" (UniqueName: \"kubernetes.io/projected/a842122e-88c2-4a73-aabf-727008e66cd0-kube-api-access-thf76\") pod \"test-model-8c684d-kserve-6899d979d5-gpqr2\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:15.588091 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.581332 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a842122e-88c2-4a73-aabf-727008e66cd0-tls-certs\") pod \"test-model-8c684d-kserve-6899d979d5-gpqr2\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:15.822815 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.822773 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" event={"ID":"b1203aec-cde6-46b4-acf9-991475721fd7","Type":"ContainerStarted","Data":"a708094d8278074c3b0f07b6dac411f847d35d734a66472059ecfd770a0f5cc9"} Nov 03 20:26:15.843687 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.843629 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" podStartSLOduration=1.8436153480000002 podStartE2EDuration="1.843615348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:15.841727295 +0000 UTC m=+1758.773594818" watchObservedRunningTime="2026-11-03 20:26:15.843615348 +0000 UTC m=+1758.775482872" Nov 03 20:26:15.874856 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:15.874818 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:16.162105 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:26:16.162064 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda842122e_88c2_4a73_aabf_727008e66cd0.slice/crio-1e4a1d4e7ad0e1ff6a758b027ffdd3bf897286a11c4fbd7445063eba175afe4f WatchSource:0}: Error finding container 1e4a1d4e7ad0e1ff6a758b027ffdd3bf897286a11c4fbd7445063eba175afe4f: Status 404 returned error can't find the container with id 1e4a1d4e7ad0e1ff6a758b027ffdd3bf897286a11c4fbd7445063eba175afe4f Nov 03 20:26:16.162560 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:16.162534 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2"] Nov 03 20:26:16.415259 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:16.415148 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:16.833059 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:16.833024 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" event={"ID":"a842122e-88c2-4a73-aabf-727008e66cd0","Type":"ContainerStarted","Data":"1d41b1e7ba7a3a17a0b7b7bd6d275e32f2733d3de3da25fea19de6bb9ee3de8f"} Nov 03 20:26:16.833239 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:16.833067 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" event={"ID":"a842122e-88c2-4a73-aabf-727008e66cd0","Type":"ContainerStarted","Data":"1e4a1d4e7ad0e1ff6a758b027ffdd3bf897286a11c4fbd7445063eba175afe4f"} Nov 03 20:26:16.857468 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:16.857417 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" podStartSLOduration=1.857403755 podStartE2EDuration="1.857403755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:16.852792715 +0000 UTC m=+1759.784660226" watchObservedRunningTime="2026-11-03 20:26:16.857403755 +0000 UTC m=+1759.789271274" Nov 03 20:26:16.875499 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:16.875468 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:17.930105 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:17.930056 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g"] Nov 03 20:26:17.930668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:17.930304 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" podUID="308f3beb-2096-41e2-8216-8b6e4a6a9e5c" containerName="istio-proxy" containerID="cri-o://dfb0f22bfa2516b278f413609d8c451fa954fccce3568fd87cf2072bb14f21fc" gracePeriod=30 Nov 03 20:26:23.227478 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.227449 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:26:23.361759 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.361715 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-kube-api-access-9ztp2\" (UniqueName: \"kubernetes.io/projected/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-kube-api-access-9ztp2\") pod \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " Nov 03 20:26:23.362882 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.362837 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istiod-ca-cert\") pod \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " Nov 03 20:26:23.363040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.362909 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-credential-socket\") pod \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " Nov 03 20:26:23.363040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.362971 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-envoy\") pod \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " Nov 03 20:26:23.363040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.363014 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-data\") pod \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " Nov 03 20:26:23.363248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.363059 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-podinfo\") pod \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " Nov 03 20:26:23.363299 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.363249 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-workload-socket\") pod \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " Nov 03 20:26:23.363708 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.363356 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-workload-certs\") pod \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " Nov 03 20:26:23.363708 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.363474 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-token\" (UniqueName: \"kubernetes.io/projected/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-token\") pod \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\" (UID: \"308f3beb-2096-41e2-8216-8b6e4a6a9e5c\") " Nov 03 20:26:23.364201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.364140 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-data" pod "308f3beb-2096-41e2-8216-8b6e4a6a9e5c" (UID: "308f3beb-2096-41e2-8216-8b6e4a6a9e5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:23.364350 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.364266 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-credential-socket" pod "308f3beb-2096-41e2-8216-8b6e4a6a9e5c" (UID: "308f3beb-2096-41e2-8216-8b6e4a6a9e5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:23.364350 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.364328 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-workload-certs" pod "308f3beb-2096-41e2-8216-8b6e4a6a9e5c" (UID: "308f3beb-2096-41e2-8216-8b6e4a6a9e5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:23.364915 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.364890 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istiod-ca-cert" pod "308f3beb-2096-41e2-8216-8b6e4a6a9e5c" (UID: "308f3beb-2096-41e2-8216-8b6e4a6a9e5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:23.365626 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.365600 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-workload-socket" pod "308f3beb-2096-41e2-8216-8b6e4a6a9e5c" (UID: "308f3beb-2096-41e2-8216-8b6e4a6a9e5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:23.366996 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.366972 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.368075 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.368049 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.368075 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.368078 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.368075 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.368092 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.369713 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.368107 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.371862 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.371825 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-envoy" pod "308f3beb-2096-41e2-8216-8b6e4a6a9e5c" (UID: "308f3beb-2096-41e2-8216-8b6e4a6a9e5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:23.371961 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.371875 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-token" pod "308f3beb-2096-41e2-8216-8b6e4a6a9e5c" (UID: "308f3beb-2096-41e2-8216-8b6e4a6a9e5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:23.372222 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.372195 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-kube-api-access-9ztp2" pod "308f3beb-2096-41e2-8216-8b6e4a6a9e5c" (UID: "308f3beb-2096-41e2-8216-8b6e4a6a9e5c"). InnerVolumeSpecName "kube-api-access-9ztp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:23.373596 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.373573 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-podinfo" pod "308f3beb-2096-41e2-8216-8b6e4a6a9e5c" (UID: "308f3beb-2096-41e2-8216-8b6e4a6a9e5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:26:23.469500 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.469417 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.469500 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.469448 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ztp2\" (UniqueName: \"kubernetes.io/projected/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-kube-api-access-9ztp2\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.469500 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.469460 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.469500 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.469469 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308f3beb-2096-41e2-8216-8b6e4a6a9e5c-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.938513 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.938464 4918 generic.go:356] "Generic (PLEG): container finished" podID="308f3beb-2096-41e2-8216-8b6e4a6a9e5c" containerID="dfb0f22bfa2516b278f413609d8c451fa954fccce3568fd87cf2072bb14f21fc" exitCode=0 Nov 03 20:26:23.941007 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.938952 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" event={"ID":"308f3beb-2096-41e2-8216-8b6e4a6a9e5c","Type":"ContainerDied","Data":"dfb0f22bfa2516b278f413609d8c451fa954fccce3568fd87cf2072bb14f21fc"} Nov 03 20:26:23.941007 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.939059 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" event={"ID":"308f3beb-2096-41e2-8216-8b6e4a6a9e5c","Type":"ContainerDied","Data":"7c421d8f93938f365e1c7f1674b223e4dec0e2294c2b4d755a250a81405abd7e"} Nov 03 20:26:23.941007 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.939096 4918 scope.go:122] "RemoveContainer" containerID="dfb0f22bfa2516b278f413609d8c451fa954fccce3568fd87cf2072bb14f21fc" Nov 03 20:26:23.941007 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.939372 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g" Nov 03 20:26:24.000817 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:23.999958 4918 scope.go:122] "RemoveContainer" containerID="dfb0f22bfa2516b278f413609d8c451fa954fccce3568fd87cf2072bb14f21fc" Nov 03 20:26:24.014007 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:26:24.002579 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb0f22bfa2516b278f413609d8c451fa954fccce3568fd87cf2072bb14f21fc\": container with ID starting with dfb0f22bfa2516b278f413609d8c451fa954fccce3568fd87cf2072bb14f21fc not found: ID does not exist" containerID="dfb0f22bfa2516b278f413609d8c451fa954fccce3568fd87cf2072bb14f21fc" Nov 03 20:26:24.014007 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:24.002627 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb0f22bfa2516b278f413609d8c451fa954fccce3568fd87cf2072bb14f21fc"} err="failed to get container status \"dfb0f22bfa2516b278f413609d8c451fa954fccce3568fd87cf2072bb14f21fc\": rpc error: code = NotFound desc = could not find container \"dfb0f22bfa2516b278f413609d8c451fa954fccce3568fd87cf2072bb14f21fc\": container with ID starting with dfb0f22bfa2516b278f413609d8c451fa954fccce3568fd87cf2072bb14f21fc not found: ID does not exist" Nov 03 20:26:24.039354 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:24.039323 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g"] Nov 03 20:26:24.047166 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:24.047134 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-fkj2g"] Nov 03 20:26:24.415398 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:24.415365 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:24.431516 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:24.431487 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:24.963154 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:24.963115 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:26:25.218071 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:25.217984 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="308f3beb-2096-41e2-8216-8b6e4a6a9e5c" path="/var/lib/kubelet/pods/308f3beb-2096-41e2-8216-8b6e4a6a9e5c/volumes" Nov 03 20:26:25.875153 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:25.875110 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:25.887987 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:25.887960 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:25.975099 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:25.975065 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:26:28.210310 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:26:28.210272 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:26:41.429927 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:41.429886 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f"] Nov 03 20:26:41.430835 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:41.430813 4918 cpu_manager.go:414] "containerMap: removing container" podUID="308f3beb-2096-41e2-8216-8b6e4a6a9e5c" containerName="istio-proxy" Nov 03 20:26:41.430835 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:41.430833 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="308f3beb-2096-41e2-8216-8b6e4a6a9e5c" containerName="istio-proxy" Nov 03 20:26:41.431028 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:41.431015 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="308f3beb-2096-41e2-8216-8b6e4a6a9e5c" containerName="istio-proxy" Nov 03 20:26:41.436699 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:41.436676 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f" Nov 03 20:26:41.445157 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:41.445121 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f"] Nov 03 20:26:41.512250 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:41.512216 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a744eb-dockercfg-rllpz\"" Nov 03 20:26:41.512769 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:41.512588 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a744eb\"" Nov 03 20:26:41.572201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:41.553110 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwfkn\" (UniqueName: \"kubernetes.io/projected/38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c-kube-api-access-zwfkn\") pod \"payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f\" (UID: \"38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f" Nov 03 20:26:41.605214 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:41.574098 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f\" (UID: \"38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f" Nov 03 20:26:41.740436 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:41.708722 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwfkn\" (UniqueName: \"kubernetes.io/projected/38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c-kube-api-access-zwfkn\") pod \"payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f\" (UID: \"38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f" Nov 03 20:26:41.740436 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:41.726856 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f\" (UID: \"38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f" Nov 03 20:26:41.740436 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:41.728164 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f\" (UID: \"38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f" Nov 03 20:26:41.769227 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:41.769189 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwfkn\" (UniqueName: \"kubernetes.io/projected/38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c-kube-api-access-zwfkn\") pod \"payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f\" (UID: \"38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f" Nov 03 20:26:42.058930 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:42.058849 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f" Nov 03 20:26:42.347913 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:26:42.347879 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b48de8_eb8c_46cc_bfa4_6ef2a2bb0a4c.slice/crio-bda811ec90e31a50acca43dcb844cf5a3238185ad657ec360158964d114627f7 WatchSource:0}: Error finding container bda811ec90e31a50acca43dcb844cf5a3238185ad657ec360158964d114627f7: Status 404 returned error can't find the container with id bda811ec90e31a50acca43dcb844cf5a3238185ad657ec360158964d114627f7 Nov 03 20:26:42.348293 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:42.348266 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f"] Nov 03 20:26:43.268256 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:43.268214 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f" event={"ID":"38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c","Type":"ContainerStarted","Data":"46a44c88a6578ce74cb13d0a6bde3e69ac8244fc33008061067dc551423581e5"} Nov 03 20:26:43.268832 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:43.268286 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f" event={"ID":"38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c","Type":"ContainerStarted","Data":"bda811ec90e31a50acca43dcb844cf5a3238185ad657ec360158964d114627f7"} Nov 03 20:26:43.268832 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:43.268340 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f" Nov 03 20:26:43.297414 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:43.297348 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f" podStartSLOduration=2.297329048 podStartE2EDuration="2.297329048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:43.291959049 +0000 UTC m=+1786.223826560" watchObservedRunningTime="2026-11-03 20:26:43.297329048 +0000 UTC m=+1786.229196559" Nov 03 20:26:43.391761 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:43.391725 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95"] Nov 03 20:26:43.397272 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:43.397245 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95" Nov 03 20:26:43.401366 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:43.401318 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-45e569-dockercfg-52dfw\"" Nov 03 20:26:43.409232 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:43.409202 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95"] Nov 03 20:26:43.481452 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:43.481421 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-45e569\"" Nov 03 20:26:43.579597 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:43.577850 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be8a9e79-44b3-4610-acbc-ee4dd69c121f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95\" (UID: \"be8a9e79-44b3-4610-acbc-ee4dd69c121f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95" Nov 03 20:26:43.590100 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:43.590059 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zssff\" (UniqueName: \"kubernetes.io/projected/be8a9e79-44b3-4610-acbc-ee4dd69c121f-kube-api-access-zssff\") pod \"payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95\" (UID: \"be8a9e79-44b3-4610-acbc-ee4dd69c121f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95" Nov 03 20:26:43.716938 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:43.716894 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be8a9e79-44b3-4610-acbc-ee4dd69c121f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95\" (UID: \"be8a9e79-44b3-4610-acbc-ee4dd69c121f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95" Nov 03 20:26:43.717383 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:43.717356 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zssff\" (UniqueName: \"kubernetes.io/projected/be8a9e79-44b3-4610-acbc-ee4dd69c121f-kube-api-access-zssff\") pod \"payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95\" (UID: \"be8a9e79-44b3-4610-acbc-ee4dd69c121f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95" Nov 03 20:26:43.719038 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:43.719013 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be8a9e79-44b3-4610-acbc-ee4dd69c121f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95\" (UID: \"be8a9e79-44b3-4610-acbc-ee4dd69c121f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95" Nov 03 20:26:43.792912 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:43.792846 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zssff\" (UniqueName: \"kubernetes.io/projected/be8a9e79-44b3-4610-acbc-ee4dd69c121f-kube-api-access-zssff\") pod \"payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95\" (UID: \"be8a9e79-44b3-4610-acbc-ee4dd69c121f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95" Nov 03 20:26:44.024685 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:44.024601 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95" Nov 03 20:26:44.313531 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:26:44.313499 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8a9e79_44b3_4610_acbc_ee4dd69c121f.slice/crio-fd796ce2e12ed7c57c09ea95b2d6556483dcd58c1ba3254652c2d8be3e66de90 WatchSource:0}: Error finding container fd796ce2e12ed7c57c09ea95b2d6556483dcd58c1ba3254652c2d8be3e66de90: Status 404 returned error can't find the container with id fd796ce2e12ed7c57c09ea95b2d6556483dcd58c1ba3254652c2d8be3e66de90 Nov 03 20:26:44.314818 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:44.314784 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95"] Nov 03 20:26:45.286722 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:45.286684 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95" event={"ID":"be8a9e79-44b3-4610-acbc-ee4dd69c121f","Type":"ContainerStarted","Data":"abc0e897275b5d77f1eaffa2d86821926b2fd783b2a911fcf82a09c4a00df8ab"} Nov 03 20:26:45.286894 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:45.286746 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95" event={"ID":"be8a9e79-44b3-4610-acbc-ee4dd69c121f","Type":"ContainerStarted","Data":"fd796ce2e12ed7c57c09ea95b2d6556483dcd58c1ba3254652c2d8be3e66de90"} Nov 03 20:26:45.286894 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:45.286770 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95" Nov 03 20:26:45.307704 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:45.307651 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95" podStartSLOduration=2.307637946 podStartE2EDuration="2.307637946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:45.304570113 +0000 UTC m=+1788.236437625" watchObservedRunningTime="2026-11-03 20:26:45.307637946 +0000 UTC m=+1788.239505456" Nov 03 20:26:54.280641 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:54.280611 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f" Nov 03 20:26:56.298765 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:56.298716 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95" Nov 03 20:26:57.471243 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:57.471209 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:26:57.476667 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:57.476647 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:26:57.621131 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:57.621097 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds"] Nov 03 20:26:57.621391 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:26:57.621355 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" podUID="da35ea39-491f-4d1b-b762-3d231a0aa1f9" containerName="istio-proxy" containerID="cri-o://469b703fc5a46fca5dd115ead8e42c7bf7be88d4785e1abe8615e63a6948ace8" gracePeriod=30 Nov 03 20:26:58.211383 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:26:58.211351 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:27:02.919004 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:02.918976 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:27:03.041039 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.040998 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-podinfo\") pod \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " Nov 03 20:27:03.041479 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.041451 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-credential-socket\") pod \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " Nov 03 20:27:03.041697 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.041678 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-token\" (UniqueName: \"kubernetes.io/projected/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-token\") pod \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " Nov 03 20:27:03.041886 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.041866 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-data\") pod \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " Nov 03 20:27:03.042061 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.042045 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da35ea39-491f-4d1b-b762-3d231a0aa1f9-kube-api-access-brvkl\" (UniqueName: \"kubernetes.io/projected/da35ea39-491f-4d1b-b762-3d231a0aa1f9-kube-api-access-brvkl\") pod \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " Nov 03 20:27:03.042200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.042148 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-data" pod "da35ea39-491f-4d1b-b762-3d231a0aa1f9" (UID: "da35ea39-491f-4d1b-b762-3d231a0aa1f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:03.043164 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.042749 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-credential-socket" pod "da35ea39-491f-4d1b-b762-3d231a0aa1f9" (UID: "da35ea39-491f-4d1b-b762-3d231a0aa1f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:03.043317 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.043296 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-envoy\") pod \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " Nov 03 20:27:03.043481 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.043463 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istiod-ca-cert\") pod \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " Nov 03 20:27:03.043616 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.043599 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-workload-certs\") pod \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " Nov 03 20:27:03.043899 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.043879 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-workload-socket\") pod \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\" (UID: \"da35ea39-491f-4d1b-b762-3d231a0aa1f9\") " Nov 03 20:27:03.044251 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.044224 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-workload-socket" pod "da35ea39-491f-4d1b-b762-3d231a0aa1f9" (UID: "da35ea39-491f-4d1b-b762-3d231a0aa1f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:03.044492 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.044464 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-token" pod "da35ea39-491f-4d1b-b762-3d231a0aa1f9" (UID: "da35ea39-491f-4d1b-b762-3d231a0aa1f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:03.044581 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.044559 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-workload-certs" pod "da35ea39-491f-4d1b-b762-3d231a0aa1f9" (UID: "da35ea39-491f-4d1b-b762-3d231a0aa1f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:03.044647 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.044578 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istiod-ca-cert" pod "da35ea39-491f-4d1b-b762-3d231a0aa1f9" (UID: "da35ea39-491f-4d1b-b762-3d231a0aa1f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:27:03.045267 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.045244 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:03.045407 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.045392 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:03.045507 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.045494 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:03.045596 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.045584 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:03.046010 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.045990 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:03.046121 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.046107 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:03.046558 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.046491 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-podinfo" pod "da35ea39-491f-4d1b-b762-3d231a0aa1f9" (UID: "da35ea39-491f-4d1b-b762-3d231a0aa1f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:27:03.046558 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.046517 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da35ea39-491f-4d1b-b762-3d231a0aa1f9-kube-api-access-brvkl" pod "da35ea39-491f-4d1b-b762-3d231a0aa1f9" (UID: "da35ea39-491f-4d1b-b762-3d231a0aa1f9"). InnerVolumeSpecName "kube-api-access-brvkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:03.046723 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.046676 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-envoy" pod "da35ea39-491f-4d1b-b762-3d231a0aa1f9" (UID: "da35ea39-491f-4d1b-b762-3d231a0aa1f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:03.149099 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.149013 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:03.149099 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.149061 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brvkl\" (UniqueName: \"kubernetes.io/projected/da35ea39-491f-4d1b-b762-3d231a0aa1f9-kube-api-access-brvkl\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:03.149099 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.149077 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da35ea39-491f-4d1b-b762-3d231a0aa1f9-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:03.467968 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.462787 4918 generic.go:356] "Generic (PLEG): container finished" podID="da35ea39-491f-4d1b-b762-3d231a0aa1f9" containerID="469b703fc5a46fca5dd115ead8e42c7bf7be88d4785e1abe8615e63a6948ace8" exitCode=0 Nov 03 20:27:03.467968 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.462911 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" event={"ID":"da35ea39-491f-4d1b-b762-3d231a0aa1f9","Type":"ContainerDied","Data":"469b703fc5a46fca5dd115ead8e42c7bf7be88d4785e1abe8615e63a6948ace8"} Nov 03 20:27:03.467968 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.462965 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" event={"ID":"da35ea39-491f-4d1b-b762-3d231a0aa1f9","Type":"ContainerDied","Data":"0aded6041af58096a6cb0e8138b5fc357e1c0bf55f2745547a7ced1bce82290d"} Nov 03 20:27:03.467968 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.462994 4918 scope.go:122] "RemoveContainer" containerID="469b703fc5a46fca5dd115ead8e42c7bf7be88d4785e1abe8615e63a6948ace8" Nov 03 20:27:03.467968 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.463165 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds" Nov 03 20:27:03.499986 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.499954 4918 scope.go:122] "RemoveContainer" containerID="469b703fc5a46fca5dd115ead8e42c7bf7be88d4785e1abe8615e63a6948ace8" Nov 03 20:27:03.505169 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:27:03.505129 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"469b703fc5a46fca5dd115ead8e42c7bf7be88d4785e1abe8615e63a6948ace8\": container with ID starting with 469b703fc5a46fca5dd115ead8e42c7bf7be88d4785e1abe8615e63a6948ace8 not found: ID does not exist" containerID="469b703fc5a46fca5dd115ead8e42c7bf7be88d4785e1abe8615e63a6948ace8" Nov 03 20:27:03.505422 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.505391 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"469b703fc5a46fca5dd115ead8e42c7bf7be88d4785e1abe8615e63a6948ace8"} err="failed to get container status \"469b703fc5a46fca5dd115ead8e42c7bf7be88d4785e1abe8615e63a6948ace8\": rpc error: code = NotFound desc = could not find container \"469b703fc5a46fca5dd115ead8e42c7bf7be88d4785e1abe8615e63a6948ace8\": container with ID starting with 469b703fc5a46fca5dd115ead8e42c7bf7be88d4785e1abe8615e63a6948ace8 not found: ID does not exist" Nov 03 20:27:03.516473 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.516445 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds"] Nov 03 20:27:03.520326 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:03.520294 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-k6gds"] Nov 03 20:27:05.221013 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:05.220975 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da35ea39-491f-4d1b-b762-3d231a0aa1f9" path="/var/lib/kubelet/pods/da35ea39-491f-4d1b-b762-3d231a0aa1f9/volumes" Nov 03 20:27:08.975203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:08.950990 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:09.214203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:09.187853 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" podUID="a842122e-88c2-4a73-aabf-727008e66cd0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.143:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:09.214203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:09.210877 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" podUID="df7b9bb4-44fb-4272-a6c7-90e71d051f46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/health\": context deadline exceeded" Nov 03 20:27:09.351199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:09.324907 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" podUID="a7765d31-50d2-4227-a7a2-dbb897077456" containerName="main" probeResult="failure" output="Get \"https://10.132.0.136:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:09.795274 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:09.752008 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" podUID="d852a17f-5c32-4496-b5d7-4a5080e6c33c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.135:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:09.812976 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:09.755426 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:09.814545 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:09.787881 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 20:27:09.836810 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:09.789876 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:27:09.864222 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:09.804567 4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded" Nov 03 20:27:09.909804 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:09.909761 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" podUID="8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:09.961165 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:09.961117 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 20:27:10.098299 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:10.093061 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" podUID="df7b9bb4-44fb-4272-a6c7-90e71d051f46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:10.193752 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:10.193703 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk" podUID="c9493241-95b1-4b6d-9290-c677b2a55000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:16.859570 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:16.859518 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj"] Nov 03 20:27:16.860474 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:16.860454 4918 cpu_manager.go:414] "containerMap: removing container" podUID="da35ea39-491f-4d1b-b762-3d231a0aa1f9" containerName="istio-proxy" Nov 03 20:27:16.860474 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:16.860475 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da35ea39-491f-4d1b-b762-3d231a0aa1f9" containerName="istio-proxy" Nov 03 20:27:16.860659 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:16.860647 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="da35ea39-491f-4d1b-b762-3d231a0aa1f9" containerName="istio-proxy" Nov 03 20:27:16.887821 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:16.887775 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj"] Nov 03 20:27:16.888086 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:16.887922 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:16.898944 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:16.898916 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d0898b-openshift-default-dockercfg-2n4dg\"" Nov 03 20:27:16.944581 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:16.944541 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-istio-data\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:16.944835 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:16.944612 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769e99e9-f468-4747-ac31-5fe11e893dce-istio-token\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:16.944835 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:16.944735 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-workload-socket\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:16.944990 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:16.944969 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9s2q\" (UniqueName: \"kubernetes.io/projected/769e99e9-f468-4747-ac31-5fe11e893dce-kube-api-access-h9s2q\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:16.945127 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:16.945100 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769e99e9-f468-4747-ac31-5fe11e893dce-istio-podinfo\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:16.945227 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:16.945213 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769e99e9-f468-4747-ac31-5fe11e893dce-istiod-ca-cert\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:16.945333 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:16.945316 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-credential-socket\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:16.945448 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:16.945401 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-workload-certs\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:16.945448 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:16.945441 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-istio-envoy\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:17.046521 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:17.046482 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-workload-certs\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:17.046685 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:17.046530 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-istio-envoy\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:17.046746 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:17.046724 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-istio-data\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:17.046809 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:17.046789 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769e99e9-f468-4747-ac31-5fe11e893dce-istio-token\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:17.046860 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:17.046844 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-workload-socket\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:17.047020 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:17.046995 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-workload-certs\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:17.047088 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:17.047066 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-istio-data\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:17.047129 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:17.047068 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9s2q\" (UniqueName: \"kubernetes.io/projected/769e99e9-f468-4747-ac31-5fe11e893dce-kube-api-access-h9s2q\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:17.047269 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:17.047173 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-workload-socket\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:17.047269 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:17.047218 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769e99e9-f468-4747-ac31-5fe11e893dce-istio-podinfo\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:17.047364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:17.047319 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769e99e9-f468-4747-ac31-5fe11e893dce-istiod-ca-cert\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:17.047364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:17.047356 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-credential-socket\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:17.047748 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:17.047723 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-credential-socket\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:17.047928 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:17.047909 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769e99e9-f468-4747-ac31-5fe11e893dce-istiod-ca-cert\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:17.049852 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:17.049827 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769e99e9-f468-4747-ac31-5fe11e893dce-istio-podinfo\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:17.063172 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:17.063144 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769e99e9-f468-4747-ac31-5fe11e893dce-istio-token\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:17.063317 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:17.063293 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9s2q\" (UniqueName: \"kubernetes.io/projected/769e99e9-f468-4747-ac31-5fe11e893dce-kube-api-access-h9s2q\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:17.081512 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:17.081481 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-istio-envoy\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:17.218478 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:17.218396 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:18.915351 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:27:18.915318 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769e99e9_f468_4747_ac31_5fe11e893dce.slice/crio-1a1d5c7ad97b35e224a670c35eec88e8af04f1f39e87cc27d781f61faff2e8ce WatchSource:0}: Error finding container 1a1d5c7ad97b35e224a670c35eec88e8af04f1f39e87cc27d781f61faff2e8ce: Status 404 returned error can't find the container with id 1a1d5c7ad97b35e224a670c35eec88e8af04f1f39e87cc27d781f61faff2e8ce Nov 03 20:27:18.915679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:18.915465 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj"] Nov 03 20:27:18.919305 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:18.919260 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:27:18.919414 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:18.919356 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:27:18.919455 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:18.919408 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:27:19.441523 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:19.441484 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" event={"ID":"769e99e9-f468-4747-ac31-5fe11e893dce","Type":"ContainerStarted","Data":"9c9821b5dab68e3bc6c65284882421a04f5832885bde022f8381bd42f82f6291"} Nov 03 20:27:19.441706 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:19.441545 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" event={"ID":"769e99e9-f468-4747-ac31-5fe11e893dce","Type":"ContainerStarted","Data":"1a1d5c7ad97b35e224a670c35eec88e8af04f1f39e87cc27d781f61faff2e8ce"} Nov 03 20:27:19.467359 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:19.467244 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" podStartSLOduration=3.467225233 podStartE2EDuration="3.467225233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:19.464969813 +0000 UTC m=+1822.396837325" watchObservedRunningTime="2026-11-03 20:27:19.467225233 +0000 UTC m=+1822.399092819" Nov 03 20:27:20.219232 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:20.219201 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:20.221430 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:20.221397 4918 patch_prober.go:29] interesting pod/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 20:27:20.221566 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:20.221452 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" podUID="769e99e9-f468-4747-ac31-5fe11e893dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 20:27:20.995092 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:20.995057 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d"] Nov 03 20:27:21.077299 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.075751 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d"] Nov 03 20:27:21.077299 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.075894 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.081846 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.080676 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6cf8ae-openshift-default-dockercfg-db88k\"" Nov 03 20:27:21.094255 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.094224 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-istio-envoy\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.094437 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.094306 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvrtd\" (UniqueName: \"kubernetes.io/projected/e925174a-652a-4443-bf13-7b979933a297-kube-api-access-cvrtd\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.094510 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.094459 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e925174a-652a-4443-bf13-7b979933a297-istiod-ca-cert\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.094573 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.094531 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-workload-certs\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.094573 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.094565 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e925174a-652a-4443-bf13-7b979933a297-istio-podinfo\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.094680 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.094608 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e925174a-652a-4443-bf13-7b979933a297-istio-token\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.094680 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.094644 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-workload-socket\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.094841 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.094817 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-credential-socket\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.095133 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.095100 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-istio-data\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.196458 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.196401 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvrtd\" (UniqueName: \"kubernetes.io/projected/e925174a-652a-4443-bf13-7b979933a297-kube-api-access-cvrtd\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.196665 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.196515 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e925174a-652a-4443-bf13-7b979933a297-istiod-ca-cert\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.196665 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.196571 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-workload-certs\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.196665 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.196608 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e925174a-652a-4443-bf13-7b979933a297-istio-podinfo\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.196665 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.196646 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e925174a-652a-4443-bf13-7b979933a297-istio-token\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.196881 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.196680 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-workload-socket\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.196881 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.196831 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-credential-socket\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.196881 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.196869 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-istio-data\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.197038 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.197010 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-istio-envoy\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.197211 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.197160 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-workload-certs\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.197493 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.197469 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-credential-socket\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.197574 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.197469 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e925174a-652a-4443-bf13-7b979933a297-istiod-ca-cert\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.197574 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.197534 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-workload-socket\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.197574 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.197546 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-istio-data\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.199438 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.199417 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e925174a-652a-4443-bf13-7b979933a297-istio-podinfo\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.199567 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.199454 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-istio-envoy\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.209557 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.209535 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e925174a-652a-4443-bf13-7b979933a297-istio-token\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.209739 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.209717 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvrtd\" (UniqueName: \"kubernetes.io/projected/e925174a-652a-4443-bf13-7b979933a297-kube-api-access-cvrtd\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.249245 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.243677 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:21.249245 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.248622 4918 patch_prober.go:29] interesting pod/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 20:27:21.249245 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.248684 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" podUID="769e99e9-f468-4747-ac31-5fe11e893dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 20:27:21.715727 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.715696 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d"] Nov 03 20:27:21.716876 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:27:21.716848 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode925174a_652a_4443_bf13_7b979933a297.slice/crio-d5441f7469dcd0ee46fc28322382a75479f6898e704fafe74d5d3a4713981fcb WatchSource:0}: Error finding container d5441f7469dcd0ee46fc28322382a75479f6898e704fafe74d5d3a4713981fcb: Status 404 returned error can't find the container with id d5441f7469dcd0ee46fc28322382a75479f6898e704fafe74d5d3a4713981fcb Nov 03 20:27:21.745424 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.745363 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:27:21.745610 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.745453 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:27:21.745610 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:21.745487 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:27:22.240249 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:22.238745 4918 patch_prober.go:29] interesting pod/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 20:27:22.240249 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:22.238817 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" podUID="769e99e9-f468-4747-ac31-5fe11e893dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 20:27:22.582832 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:22.582781 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" event={"ID":"e925174a-652a-4443-bf13-7b979933a297","Type":"ContainerStarted","Data":"d276de9b945efeb594e600ea2653f8663b24eb1361b8d6a361630e88cdb3f249"} Nov 03 20:27:22.583260 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:22.582864 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" event={"ID":"e925174a-652a-4443-bf13-7b979933a297","Type":"ContainerStarted","Data":"d5441f7469dcd0ee46fc28322382a75479f6898e704fafe74d5d3a4713981fcb"} Nov 03 20:27:22.608518 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:22.608468 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" podStartSLOduration=2.608452387 podStartE2EDuration="2.608452387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:22.606629347 +0000 UTC m=+1825.538496858" watchObservedRunningTime="2026-11-03 20:27:22.608452387 +0000 UTC m=+1825.540319898" Nov 03 20:27:23.242247 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:23.240592 4918 patch_prober.go:29] interesting pod/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 20:27:23.242247 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:23.240673 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" podUID="769e99e9-f468-4747-ac31-5fe11e893dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 20:27:23.270421 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:23.270291 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:23.271282 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:23.270605 4918 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:27:23.271282 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:23.270651 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" podUID="e925174a-652a-4443-bf13-7b979933a297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:27:24.229034 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:24.227165 4918 patch_prober.go:29] interesting pod/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 20:27:24.229034 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:24.227269 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" podUID="769e99e9-f468-4747-ac31-5fe11e893dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 20:27:24.249036 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:24.247323 4918 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:27:24.249036 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:24.247396 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" podUID="e925174a-652a-4443-bf13-7b979933a297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:27:25.244230 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:25.238643 4918 patch_prober.go:29] interesting pod/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 20:27:25.244230 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:25.238721 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" podUID="769e99e9-f468-4747-ac31-5fe11e893dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 20:27:25.262609 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:25.258110 4918 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:27:25.262609 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:25.258168 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" podUID="e925174a-652a-4443-bf13-7b979933a297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:27:26.223274 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:26.222593 4918 patch_prober.go:29] interesting pod/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 20:27:26.223274 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:26.222679 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" podUID="769e99e9-f468-4747-ac31-5fe11e893dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 20:27:26.250202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:26.249606 4918 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:27:26.250202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:26.249685 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" podUID="e925174a-652a-4443-bf13-7b979933a297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:27:27.249591 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:27.249551 4918 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:27:27.249842 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:27.249616 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" podUID="e925174a-652a-4443-bf13-7b979933a297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:27:27.257395 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:27.257363 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:27.257891 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:27.257867 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:27.259712 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:27.259686 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:27:27.700266 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:27.699460 4918 generic.go:356] "Generic (PLEG): container finished" podID="3e465e5f-cb99-475a-848a-a182ceccc56e" containerID="b86ad3a64e81e11414b069a145f97d32fe73d2ef95172bb327f773f85d26732c" exitCode=0 Nov 03 20:27:27.701208 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:27.700706 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" event={"ID":"3e465e5f-cb99-475a-848a-a182ceccc56e","Type":"ContainerDied","Data":"b86ad3a64e81e11414b069a145f97d32fe73d2ef95172bb327f773f85d26732c"} Nov 03 20:27:27.701347 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:27.701215 4918 scope.go:122] "RemoveContainer" containerID="b86ad3a64e81e11414b069a145f97d32fe73d2ef95172bb327f773f85d26732c" Nov 03 20:27:27.725666 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:27.720202 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:27:27.725666 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:27.720312 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:27:27.725666 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:27.720362 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:27:28.249658 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:28.249568 4918 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:27:28.249955 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:28.249922 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" podUID="e925174a-652a-4443-bf13-7b979933a297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:27:28.263325 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:27:28.263274 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:27:28.487724 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:28.487679 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:27:28.720381 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:28.720333 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" event={"ID":"3e465e5f-cb99-475a-848a-a182ceccc56e","Type":"ContainerStarted","Data":"2dad895aa9ccdc55828c60f4e9f663fe0e96535c05dfd56982ea4b27fd257859"} Nov 03 20:27:29.266294 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:29.266256 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:29.267725 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:29.267697 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:29.269773 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:29.269732 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:27:29.530261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:29.526513 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:27:29.551207 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:29.544344 4918 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 20:27:29.551207 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:29.544425 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" podUID="3e465e5f-cb99-475a-848a-a182ceccc56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 20:27:30.525931 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:30.525896 4918 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 20:27:30.538104 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:30.538060 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" podUID="3e465e5f-cb99-475a-848a-a182ceccc56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 20:27:31.532639 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:31.532597 4918 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 20:27:31.533282 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:31.533243 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" podUID="3e465e5f-cb99-475a-848a-a182ceccc56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 20:27:32.540210 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:32.539282 4918 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 20:27:32.540210 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:32.539362 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" podUID="3e465e5f-cb99-475a-848a-a182ceccc56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 20:27:33.539200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:33.538299 4918 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 20:27:33.539200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:33.538365 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" podUID="3e465e5f-cb99-475a-848a-a182ceccc56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 20:27:34.538770 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:34.538732 4918 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 20:27:34.539457 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:34.539414 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" podUID="3e465e5f-cb99-475a-848a-a182ceccc56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 20:27:35.531091 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:35.531058 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:27:35.531626 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:35.531606 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:27:35.532343 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:27:35.532321 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:27:58.211940 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:27:58.211905 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:28:00.574229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:00.574157 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld"] Nov 03 20:28:11.120415 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:11.120363 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv"] Nov 03 20:28:15.616374 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:15.616328 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" podUID="d852a17f-5c32-4496-b5d7-4a5080e6c33c" containerName="main" containerID="cri-o://8079317e89973ae7894c616027f5147bfbed7aaa5b47ebe923d13bf75c74e3b1" gracePeriod=45 Nov 03 20:28:15.987373 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:15.978403 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:28:16.068092 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.068056 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-dshm\") pod \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " Nov 03 20:28:16.068303 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.068237 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-tmp-dir\") pod \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " Nov 03 20:28:16.068460 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.068420 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-model-cache\") pod \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " Nov 03 20:28:16.068566 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.068472 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d852a17f-5c32-4496-b5d7-4a5080e6c33c-kube-api-access-t77sp\" (UniqueName: \"kubernetes.io/projected/d852a17f-5c32-4496-b5d7-4a5080e6c33c-kube-api-access-t77sp\") pod \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " Nov 03 20:28:16.068635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.068607 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-home\" (UniqueName: \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-home\") pod \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " Nov 03 20:28:16.068635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.068615 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-tmp-dir" pod "d852a17f-5c32-4496-b5d7-4a5080e6c33c" (UID: "d852a17f-5c32-4496-b5d7-4a5080e6c33c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:16.068749 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.068656 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d852a17f-5c32-4496-b5d7-4a5080e6c33c-tls-certs\" (UniqueName: \"kubernetes.io/secret/d852a17f-5c32-4496-b5d7-4a5080e6c33c-tls-certs\") pod \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\" (UID: \"d852a17f-5c32-4496-b5d7-4a5080e6c33c\") " Nov 03 20:28:16.068868 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.068846 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-home" pod "d852a17f-5c32-4496-b5d7-4a5080e6c33c" (UID: "d852a17f-5c32-4496-b5d7-4a5080e6c33c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:16.068946 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.068914 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-model-cache" pod "d852a17f-5c32-4496-b5d7-4a5080e6c33c" (UID: "d852a17f-5c32-4496-b5d7-4a5080e6c33c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:16.069697 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.069670 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-home\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:16.069815 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.069699 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-tmp-dir\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:16.069815 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.069715 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-model-cache\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:16.071040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.071009 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d852a17f-5c32-4496-b5d7-4a5080e6c33c-kube-api-access-t77sp" pod "d852a17f-5c32-4496-b5d7-4a5080e6c33c" (UID: "d852a17f-5c32-4496-b5d7-4a5080e6c33c"). InnerVolumeSpecName "kube-api-access-t77sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:28:16.071754 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.071729 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d852a17f-5c32-4496-b5d7-4a5080e6c33c-tls-certs" pod "d852a17f-5c32-4496-b5d7-4a5080e6c33c" (UID: "d852a17f-5c32-4496-b5d7-4a5080e6c33c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:28:16.072253 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.072233 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-dshm" pod "d852a17f-5c32-4496-b5d7-4a5080e6c33c" (UID: "d852a17f-5c32-4496-b5d7-4a5080e6c33c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:16.171746 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.171660 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d852a17f-5c32-4496-b5d7-4a5080e6c33c-dshm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:16.171746 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.171693 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t77sp\" (UniqueName: \"kubernetes.io/projected/d852a17f-5c32-4496-b5d7-4a5080e6c33c-kube-api-access-t77sp\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:16.171746 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.171707 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d852a17f-5c32-4496-b5d7-4a5080e6c33c-tls-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:16.453975 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.453899 4918 generic.go:356] "Generic (PLEG): container finished" podID="d852a17f-5c32-4496-b5d7-4a5080e6c33c" containerID="8079317e89973ae7894c616027f5147bfbed7aaa5b47ebe923d13bf75c74e3b1" exitCode=0 Nov 03 20:28:16.453975 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.453938 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" event={"ID":"d852a17f-5c32-4496-b5d7-4a5080e6c33c","Type":"ContainerDied","Data":"8079317e89973ae7894c616027f5147bfbed7aaa5b47ebe923d13bf75c74e3b1"} Nov 03 20:28:16.453975 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.453951 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" Nov 03 20:28:16.453975 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.453966 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld" event={"ID":"d852a17f-5c32-4496-b5d7-4a5080e6c33c","Type":"ContainerDied","Data":"106a16f3c1d00590d22af83f71ba19d74b2e3d1e8fa639e4fe4bab846b6f9558"} Nov 03 20:28:16.454308 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.453981 4918 scope.go:122] "RemoveContainer" containerID="8079317e89973ae7894c616027f5147bfbed7aaa5b47ebe923d13bf75c74e3b1" Nov 03 20:28:16.466735 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.466715 4918 scope.go:122] "RemoveContainer" containerID="8079317e89973ae7894c616027f5147bfbed7aaa5b47ebe923d13bf75c74e3b1" Nov 03 20:28:16.467067 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:28:16.467044 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8079317e89973ae7894c616027f5147bfbed7aaa5b47ebe923d13bf75c74e3b1\": container with ID starting with 8079317e89973ae7894c616027f5147bfbed7aaa5b47ebe923d13bf75c74e3b1 not found: ID does not exist" containerID="8079317e89973ae7894c616027f5147bfbed7aaa5b47ebe923d13bf75c74e3b1" Nov 03 20:28:16.467153 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.467078 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8079317e89973ae7894c616027f5147bfbed7aaa5b47ebe923d13bf75c74e3b1"} err="failed to get container status \"8079317e89973ae7894c616027f5147bfbed7aaa5b47ebe923d13bf75c74e3b1\": rpc error: code = NotFound desc = could not find container \"8079317e89973ae7894c616027f5147bfbed7aaa5b47ebe923d13bf75c74e3b1\": container with ID starting with 8079317e89973ae7894c616027f5147bfbed7aaa5b47ebe923d13bf75c74e3b1 not found: ID does not exist" Nov 03 20:28:16.480989 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.480966 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld"] Nov 03 20:28:16.485452 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:16.485430 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-5de824/test-model-5de824-kserve-78c7f5bbd7-xpdld"] Nov 03 20:28:17.218530 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:17.218498 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d852a17f-5c32-4496-b5d7-4a5080e6c33c" path="/var/lib/kubelet/pods/d852a17f-5c32-4496-b5d7-4a5080e6c33c/volumes" Nov 03 20:28:21.616389 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:21.616349 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d"] Nov 03 20:28:21.629453 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:21.629420 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2"] Nov 03 20:28:26.147590 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.147541 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" podUID="a7765d31-50d2-4227-a7a2-dbb897077456" containerName="main" containerID="cri-o://5bddbd9625f4c4c22032a58aa6ad9cfeb33b6bcd7e9ea5976112ccd806ae25ae" gracePeriod=45 Nov 03 20:28:26.208203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.208152 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" podUID="a7765d31-50d2-4227-a7a2-dbb897077456" containerName="main" probeResult="failure" output="Get \"https://10.132.0.136:8000/ready\": dial tcp 10.132.0.136:8000: connect: connection refused" Nov 03 20:28:26.463276 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.463250 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:28:26.601222 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.594123 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-dshm\") pod \"a7765d31-50d2-4227-a7a2-dbb897077456\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " Nov 03 20:28:26.601222 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.594241 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-tmp-dir\") pod \"a7765d31-50d2-4227-a7a2-dbb897077456\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " Nov 03 20:28:26.601222 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.594315 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-model-cache\") pod \"a7765d31-50d2-4227-a7a2-dbb897077456\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " Nov 03 20:28:26.601222 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.594417 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-home\" (UniqueName: \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-home\") pod \"a7765d31-50d2-4227-a7a2-dbb897077456\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " Nov 03 20:28:26.601222 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.594551 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7765d31-50d2-4227-a7a2-dbb897077456-kube-api-access-j9bwh\" (UniqueName: \"kubernetes.io/projected/a7765d31-50d2-4227-a7a2-dbb897077456-kube-api-access-j9bwh\") pod \"a7765d31-50d2-4227-a7a2-dbb897077456\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " Nov 03 20:28:26.601222 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.594650 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7765d31-50d2-4227-a7a2-dbb897077456-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7765d31-50d2-4227-a7a2-dbb897077456-tls-certs\") pod \"a7765d31-50d2-4227-a7a2-dbb897077456\" (UID: \"a7765d31-50d2-4227-a7a2-dbb897077456\") " Nov 03 20:28:26.601222 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.598446 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-model-cache" pod "a7765d31-50d2-4227-a7a2-dbb897077456" (UID: "a7765d31-50d2-4227-a7a2-dbb897077456"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:26.601222 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.599533 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-home" pod "a7765d31-50d2-4227-a7a2-dbb897077456" (UID: "a7765d31-50d2-4227-a7a2-dbb897077456"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:26.608190 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.608142 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-tmp-dir" pod "a7765d31-50d2-4227-a7a2-dbb897077456" (UID: "a7765d31-50d2-4227-a7a2-dbb897077456"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:26.610197 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.610150 4918 generic.go:356] "Generic (PLEG): container finished" podID="a7765d31-50d2-4227-a7a2-dbb897077456" containerID="5bddbd9625f4c4c22032a58aa6ad9cfeb33b6bcd7e9ea5976112ccd806ae25ae" exitCode=0 Nov 03 20:28:26.610632 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.610605 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" event={"ID":"a7765d31-50d2-4227-a7a2-dbb897077456","Type":"ContainerDied","Data":"5bddbd9625f4c4c22032a58aa6ad9cfeb33b6bcd7e9ea5976112ccd806ae25ae"} Nov 03 20:28:26.610787 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.610769 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" event={"ID":"a7765d31-50d2-4227-a7a2-dbb897077456","Type":"ContainerDied","Data":"340fc15ed77f5905677927c297b8630b9fb8ebd71ee572ad446633f64f046131"} Nov 03 20:28:26.610921 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.610904 4918 scope.go:122] "RemoveContainer" containerID="5bddbd9625f4c4c22032a58aa6ad9cfeb33b6bcd7e9ea5976112ccd806ae25ae" Nov 03 20:28:26.611216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.611195 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv" Nov 03 20:28:26.616141 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.616103 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7765d31-50d2-4227-a7a2-dbb897077456-tls-certs" pod "a7765d31-50d2-4227-a7a2-dbb897077456" (UID: "a7765d31-50d2-4227-a7a2-dbb897077456"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:28:26.617120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.617094 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d" podUID="26102cd8-e567-4e90-acf5-a57fc2b0dae4" containerName="payload-processing" containerID="cri-o://fd355472123a5b5cb277200bc340c149b1ae776ae0ddf6dd2932f5f3b3720e5e" gracePeriod=25 Nov 03 20:28:26.618479 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.618450 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-dshm" pod "a7765d31-50d2-4227-a7a2-dbb897077456" (UID: "a7765d31-50d2-4227-a7a2-dbb897077456"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:26.618629 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.618501 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7765d31-50d2-4227-a7a2-dbb897077456-kube-api-access-j9bwh" pod "a7765d31-50d2-4227-a7a2-dbb897077456" (UID: "a7765d31-50d2-4227-a7a2-dbb897077456"). InnerVolumeSpecName "kube-api-access-j9bwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:28:26.630593 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.630556 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2" podUID="3f3259f2-a529-4ea7-a637-b39c29e0a72f" containerName="payload-pre-processing" containerID="cri-o://734e2b83b1aba2a48a01ef2271cbae38cb988944dadd60d6ebfbe2164e81bc75" gracePeriod=25 Nov 03 20:28:26.698112 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.698076 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-dshm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:26.698260 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.698113 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-tmp-dir\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:26.698260 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.698128 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-model-cache\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:26.698260 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.698145 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7765d31-50d2-4227-a7a2-dbb897077456-home\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:26.698260 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.698160 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9bwh\" (UniqueName: \"kubernetes.io/projected/a7765d31-50d2-4227-a7a2-dbb897077456-kube-api-access-j9bwh\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:26.698260 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.698219 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7765d31-50d2-4227-a7a2-dbb897077456-tls-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:26.760059 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.760018 4918 scope.go:122] "RemoveContainer" containerID="5bddbd9625f4c4c22032a58aa6ad9cfeb33b6bcd7e9ea5976112ccd806ae25ae" Nov 03 20:28:26.760530 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:28:26.760497 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bddbd9625f4c4c22032a58aa6ad9cfeb33b6bcd7e9ea5976112ccd806ae25ae\": container with ID starting with 5bddbd9625f4c4c22032a58aa6ad9cfeb33b6bcd7e9ea5976112ccd806ae25ae not found: ID does not exist" containerID="5bddbd9625f4c4c22032a58aa6ad9cfeb33b6bcd7e9ea5976112ccd806ae25ae" Nov 03 20:28:26.760639 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.760536 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bddbd9625f4c4c22032a58aa6ad9cfeb33b6bcd7e9ea5976112ccd806ae25ae"} err="failed to get container status \"5bddbd9625f4c4c22032a58aa6ad9cfeb33b6bcd7e9ea5976112ccd806ae25ae\": rpc error: code = NotFound desc = could not find container \"5bddbd9625f4c4c22032a58aa6ad9cfeb33b6bcd7e9ea5976112ccd806ae25ae\": container with ID starting with 5bddbd9625f4c4c22032a58aa6ad9cfeb33b6bcd7e9ea5976112ccd806ae25ae not found: ID does not exist" Nov 03 20:28:26.911088 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.911050 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d" Nov 03 20:28:26.952925 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.952838 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv"] Nov 03 20:28:26.962964 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.962932 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d9f082/test-model-d9f082-kserve-57687df945-6l8wv"] Nov 03 20:28:26.987560 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:26.987530 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2" Nov 03 20:28:27.002423 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.002390 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26102cd8-e567-4e90-acf5-a57fc2b0dae4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26102cd8-e567-4e90-acf5-a57fc2b0dae4-plugins-config-volume\") pod \"26102cd8-e567-4e90-acf5-a57fc2b0dae4\" (UID: \"26102cd8-e567-4e90-acf5-a57fc2b0dae4\") " Nov 03 20:28:27.002605 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.002457 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26102cd8-e567-4e90-acf5-a57fc2b0dae4-kube-api-access-r7g25\" (UniqueName: \"kubernetes.io/projected/26102cd8-e567-4e90-acf5-a57fc2b0dae4-kube-api-access-r7g25\") pod \"26102cd8-e567-4e90-acf5-a57fc2b0dae4\" (UID: \"26102cd8-e567-4e90-acf5-a57fc2b0dae4\") " Nov 03 20:28:27.002828 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.002789 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26102cd8-e567-4e90-acf5-a57fc2b0dae4-plugins-config-volume" pod "26102cd8-e567-4e90-acf5-a57fc2b0dae4" (UID: "26102cd8-e567-4e90-acf5-a57fc2b0dae4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:28:27.003340 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.003318 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26102cd8-e567-4e90-acf5-a57fc2b0dae4-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:27.004792 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.004768 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26102cd8-e567-4e90-acf5-a57fc2b0dae4-kube-api-access-r7g25" pod "26102cd8-e567-4e90-acf5-a57fc2b0dae4" (UID: "26102cd8-e567-4e90-acf5-a57fc2b0dae4"). InnerVolumeSpecName "kube-api-access-r7g25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:28:27.104372 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.104340 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f3259f2-a529-4ea7-a637-b39c29e0a72f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3259f2-a529-4ea7-a637-b39c29e0a72f-plugins-config-volume\") pod \"3f3259f2-a529-4ea7-a637-b39c29e0a72f\" (UID: \"3f3259f2-a529-4ea7-a637-b39c29e0a72f\") " Nov 03 20:28:27.104604 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.104586 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3259f2-a529-4ea7-a637-b39c29e0a72f-kube-api-access-bz8xp\" (UniqueName: \"kubernetes.io/projected/3f3259f2-a529-4ea7-a637-b39c29e0a72f-kube-api-access-bz8xp\") pod \"3f3259f2-a529-4ea7-a637-b39c29e0a72f\" (UID: \"3f3259f2-a529-4ea7-a637-b39c29e0a72f\") " Nov 03 20:28:27.104703 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.104679 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3259f2-a529-4ea7-a637-b39c29e0a72f-plugins-config-volume" pod "3f3259f2-a529-4ea7-a637-b39c29e0a72f" (UID: "3f3259f2-a529-4ea7-a637-b39c29e0a72f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:28:27.105314 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.105297 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7g25\" (UniqueName: \"kubernetes.io/projected/26102cd8-e567-4e90-acf5-a57fc2b0dae4-kube-api-access-r7g25\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:27.105381 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.105325 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3259f2-a529-4ea7-a637-b39c29e0a72f-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:27.106707 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.106686 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3259f2-a529-4ea7-a637-b39c29e0a72f-kube-api-access-bz8xp" pod "3f3259f2-a529-4ea7-a637-b39c29e0a72f" (UID: "3f3259f2-a529-4ea7-a637-b39c29e0a72f"). InnerVolumeSpecName "kube-api-access-bz8xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:28:27.206945 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.206864 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz8xp\" (UniqueName: \"kubernetes.io/projected/3f3259f2-a529-4ea7-a637-b39c29e0a72f-kube-api-access-bz8xp\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:27.217517 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.217486 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7765d31-50d2-4227-a7a2-dbb897077456" path="/var/lib/kubelet/pods/a7765d31-50d2-4227-a7a2-dbb897077456/volumes" Nov 03 20:28:27.619570 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.619532 4918 generic.go:356] "Generic (PLEG): container finished" podID="26102cd8-e567-4e90-acf5-a57fc2b0dae4" containerID="fd355472123a5b5cb277200bc340c149b1ae776ae0ddf6dd2932f5f3b3720e5e" exitCode=0 Nov 03 20:28:27.619802 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.619587 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d" Nov 03 20:28:27.619802 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.619626 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d" event={"ID":"26102cd8-e567-4e90-acf5-a57fc2b0dae4","Type":"ContainerDied","Data":"fd355472123a5b5cb277200bc340c149b1ae776ae0ddf6dd2932f5f3b3720e5e"} Nov 03 20:28:27.619802 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.619687 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d" event={"ID":"26102cd8-e567-4e90-acf5-a57fc2b0dae4","Type":"ContainerDied","Data":"1c97d78ca394db084b7b987372ea4c5cfe8cf4f405c996bca171926d3786184f"} Nov 03 20:28:27.619802 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.619719 4918 scope.go:122] "RemoveContainer" containerID="fd355472123a5b5cb277200bc340c149b1ae776ae0ddf6dd2932f5f3b3720e5e" Nov 03 20:28:27.621096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.621070 4918 generic.go:356] "Generic (PLEG): container finished" podID="3f3259f2-a529-4ea7-a637-b39c29e0a72f" containerID="734e2b83b1aba2a48a01ef2271cbae38cb988944dadd60d6ebfbe2164e81bc75" exitCode=0 Nov 03 20:28:27.621239 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.621150 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2" Nov 03 20:28:27.621239 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.621189 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2" event={"ID":"3f3259f2-a529-4ea7-a637-b39c29e0a72f","Type":"ContainerDied","Data":"734e2b83b1aba2a48a01ef2271cbae38cb988944dadd60d6ebfbe2164e81bc75"} Nov 03 20:28:27.621360 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.621238 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2" event={"ID":"3f3259f2-a529-4ea7-a637-b39c29e0a72f","Type":"ContainerDied","Data":"4d30d9fdb0012f41bde9f6a9f6861551357dbe69c5cb71efde89d158108a92b7"} Nov 03 20:28:27.631869 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.631848 4918 scope.go:122] "RemoveContainer" containerID="fd355472123a5b5cb277200bc340c149b1ae776ae0ddf6dd2932f5f3b3720e5e" Nov 03 20:28:27.632129 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:28:27.632111 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd355472123a5b5cb277200bc340c149b1ae776ae0ddf6dd2932f5f3b3720e5e\": container with ID starting with fd355472123a5b5cb277200bc340c149b1ae776ae0ddf6dd2932f5f3b3720e5e not found: ID does not exist" containerID="fd355472123a5b5cb277200bc340c149b1ae776ae0ddf6dd2932f5f3b3720e5e" Nov 03 20:28:27.632198 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.632140 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd355472123a5b5cb277200bc340c149b1ae776ae0ddf6dd2932f5f3b3720e5e"} err="failed to get container status \"fd355472123a5b5cb277200bc340c149b1ae776ae0ddf6dd2932f5f3b3720e5e\": rpc error: code = NotFound desc = could not find container \"fd355472123a5b5cb277200bc340c149b1ae776ae0ddf6dd2932f5f3b3720e5e\": container with ID starting with fd355472123a5b5cb277200bc340c149b1ae776ae0ddf6dd2932f5f3b3720e5e not found: ID does not exist" Nov 03 20:28:27.632198 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.632157 4918 scope.go:122] "RemoveContainer" containerID="734e2b83b1aba2a48a01ef2271cbae38cb988944dadd60d6ebfbe2164e81bc75" Nov 03 20:28:27.648858 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.648829 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2"] Nov 03 20:28:27.649012 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.648908 4918 scope.go:122] "RemoveContainer" containerID="734e2b83b1aba2a48a01ef2271cbae38cb988944dadd60d6ebfbe2164e81bc75" Nov 03 20:28:27.649492 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:28:27.649468 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"734e2b83b1aba2a48a01ef2271cbae38cb988944dadd60d6ebfbe2164e81bc75\": container with ID starting with 734e2b83b1aba2a48a01ef2271cbae38cb988944dadd60d6ebfbe2164e81bc75 not found: ID does not exist" containerID="734e2b83b1aba2a48a01ef2271cbae38cb988944dadd60d6ebfbe2164e81bc75" Nov 03 20:28:27.649587 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.649498 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"734e2b83b1aba2a48a01ef2271cbae38cb988944dadd60d6ebfbe2164e81bc75"} err="failed to get container status \"734e2b83b1aba2a48a01ef2271cbae38cb988944dadd60d6ebfbe2164e81bc75\": rpc error: code = NotFound desc = could not find container \"734e2b83b1aba2a48a01ef2271cbae38cb988944dadd60d6ebfbe2164e81bc75\": container with ID starting with 734e2b83b1aba2a48a01ef2271cbae38cb988944dadd60d6ebfbe2164e81bc75 not found: ID does not exist" Nov 03 20:28:27.650777 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.650756 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5de824-cd57ccb7f-lfss2"] Nov 03 20:28:27.662449 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.662422 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d"] Nov 03 20:28:27.666793 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:27.666763 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5de824-7c769ff464-2bj5d"] Nov 03 20:28:28.211231 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:28:28.211160 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:28:29.218303 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:29.218272 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26102cd8-e567-4e90-acf5-a57fc2b0dae4" path="/var/lib/kubelet/pods/26102cd8-e567-4e90-acf5-a57fc2b0dae4/volumes" Nov 03 20:28:29.218989 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:29.218767 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f3259f2-a529-4ea7-a637-b39c29e0a72f" path="/var/lib/kubelet/pods/3f3259f2-a529-4ea7-a637-b39c29e0a72f/volumes" Nov 03 20:28:35.507691 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.507649 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv"] Nov 03 20:28:35.508813 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.508788 4918 cpu_manager.go:414] "containerMap: removing container" podUID="a7765d31-50d2-4227-a7a2-dbb897077456" containerName="main" Nov 03 20:28:35.508813 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.508812 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7765d31-50d2-4227-a7a2-dbb897077456" containerName="main" Nov 03 20:28:35.509013 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.508848 4918 cpu_manager.go:414] "containerMap: removing container" podUID="26102cd8-e567-4e90-acf5-a57fc2b0dae4" containerName="payload-processing" Nov 03 20:28:35.509013 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.508856 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26102cd8-e567-4e90-acf5-a57fc2b0dae4" containerName="payload-processing" Nov 03 20:28:35.509013 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.508882 4918 cpu_manager.go:414] "containerMap: removing container" podUID="d852a17f-5c32-4496-b5d7-4a5080e6c33c" containerName="main" Nov 03 20:28:35.509013 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.508890 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d852a17f-5c32-4496-b5d7-4a5080e6c33c" containerName="main" Nov 03 20:28:35.509013 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.508907 4918 cpu_manager.go:414] "containerMap: removing container" podUID="3f3259f2-a529-4ea7-a637-b39c29e0a72f" containerName="payload-pre-processing" Nov 03 20:28:35.509013 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.508915 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f3259f2-a529-4ea7-a637-b39c29e0a72f" containerName="payload-pre-processing" Nov 03 20:28:35.509381 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.509136 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7765d31-50d2-4227-a7a2-dbb897077456" containerName="main" Nov 03 20:28:35.509381 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.509155 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="26102cd8-e567-4e90-acf5-a57fc2b0dae4" containerName="payload-processing" Nov 03 20:28:35.509381 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.509188 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d852a17f-5c32-4496-b5d7-4a5080e6c33c" containerName="main" Nov 03 20:28:35.509381 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.509199 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f3259f2-a529-4ea7-a637-b39c29e0a72f" containerName="payload-pre-processing" Nov 03 20:28:35.515047 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.515024 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:35.527367 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.527348 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-45e569\"/\"test-model-45e569-kserve-self-signed-certs\"" Nov 03 20:28:35.528777 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.528752 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-45e569\"/\"default-dockercfg-7fr5f\"" Nov 03 20:28:35.528910 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.528896 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-45e569\"/\"openshift-service-ca.crt\"" Nov 03 20:28:35.528990 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.528977 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-45e569\"/\"kube-root-ca.crt\"" Nov 03 20:28:35.539798 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.539769 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv"] Nov 03 20:28:35.646146 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.646098 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-home\") pod \"test-model-45e569-kserve-849bdb9979-nf7cv\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:35.646384 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.646256 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hqlb\" (UniqueName: \"kubernetes.io/projected/54a415e5-f853-4579-9f3f-0c3c7a9216e5-kube-api-access-6hqlb\") pod \"test-model-45e569-kserve-849bdb9979-nf7cv\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:35.646384 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.646290 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-tmp-dir\") pod \"test-model-45e569-kserve-849bdb9979-nf7cv\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:35.646467 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.646428 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54a415e5-f853-4579-9f3f-0c3c7a9216e5-tls-certs\") pod \"test-model-45e569-kserve-849bdb9979-nf7cv\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:35.646559 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.646538 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-model-cache\") pod \"test-model-45e569-kserve-849bdb9979-nf7cv\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:35.646690 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.646673 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-dshm\") pod \"test-model-45e569-kserve-849bdb9979-nf7cv\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:35.748409 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.748335 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-model-cache\") pod \"test-model-45e569-kserve-849bdb9979-nf7cv\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:35.750210 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.749272 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-dshm\") pod \"test-model-45e569-kserve-849bdb9979-nf7cv\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:35.751438 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.750916 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-home\") pod \"test-model-45e569-kserve-849bdb9979-nf7cv\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:35.751697 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.751672 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hqlb\" (UniqueName: \"kubernetes.io/projected/54a415e5-f853-4579-9f3f-0c3c7a9216e5-kube-api-access-6hqlb\") pod \"test-model-45e569-kserve-849bdb9979-nf7cv\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:35.752449 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.752429 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-tmp-dir\") pod \"test-model-45e569-kserve-849bdb9979-nf7cv\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:35.752734 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.752717 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54a415e5-f853-4579-9f3f-0c3c7a9216e5-tls-certs\") pod \"test-model-45e569-kserve-849bdb9979-nf7cv\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:35.753689 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.753666 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-tmp-dir\") pod \"test-model-45e569-kserve-849bdb9979-nf7cv\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:35.753827 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.749074 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-model-cache\") pod \"test-model-45e569-kserve-849bdb9979-nf7cv\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:35.753930 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.752071 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-home\") pod \"test-model-45e569-kserve-849bdb9979-nf7cv\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:35.756675 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.756654 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54a415e5-f853-4579-9f3f-0c3c7a9216e5-tls-certs\") pod \"test-model-45e569-kserve-849bdb9979-nf7cv\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:35.796243 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.795602 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-dshm\") pod \"test-model-45e569-kserve-849bdb9979-nf7cv\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:35.803197 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.801737 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hqlb\" (UniqueName: \"kubernetes.io/projected/54a415e5-f853-4579-9f3f-0c3c7a9216e5-kube-api-access-6hqlb\") pod \"test-model-45e569-kserve-849bdb9979-nf7cv\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:35.834944 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:35.834631 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:36.139430 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:36.139398 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv"] Nov 03 20:28:36.140939 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:28:36.140899 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54a415e5_f853_4579_9f3f_0c3c7a9216e5.slice/crio-550172d04605202789dc715958a0097ed69ae10d8a27dc04dddf1cfc97e04417 WatchSource:0}: Error finding container 550172d04605202789dc715958a0097ed69ae10d8a27dc04dddf1cfc97e04417: Status 404 returned error can't find the container with id 550172d04605202789dc715958a0097ed69ae10d8a27dc04dddf1cfc97e04417 Nov 03 20:28:36.798657 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:36.798617 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" event={"ID":"54a415e5-f853-4579-9f3f-0c3c7a9216e5","Type":"ContainerStarted","Data":"35e5a6417f38e1beeb118049bb5e6fff6b066320581db3af8688bef398ba2e62"} Nov 03 20:28:36.799123 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:36.798679 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" event={"ID":"54a415e5-f853-4579-9f3f-0c3c7a9216e5","Type":"ContainerStarted","Data":"550172d04605202789dc715958a0097ed69ae10d8a27dc04dddf1cfc97e04417"} Nov 03 20:28:36.821122 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:36.821076 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" podStartSLOduration=1.821063029 podStartE2EDuration="1.821063029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:36.817800195 +0000 UTC m=+1899.749667728" watchObservedRunningTime="2026-11-03 20:28:36.821063029 +0000 UTC m=+1899.752930541" Nov 03 20:28:36.836836 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:36.836809 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:43.230380 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:43.230264 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt"] Nov 03 20:28:45.836531 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:45.836497 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:45.849695 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:45.849665 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:45.892853 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:45.892811 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:28:48.230869 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:48.230811 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt" podUID="b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f" containerName="payload-processing" containerID="cri-o://ac9e9de6f4bf9505f9481047a3da48824dd080196c8bd9beb364e0bb2f971dc6" gracePeriod=25 Nov 03 20:28:48.562983 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:48.562954 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt" Nov 03 20:28:48.692820 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:48.692791 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f-kube-api-access-487pv\" (UniqueName: \"kubernetes.io/projected/b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f-kube-api-access-487pv\") pod \"b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f\" (UID: \"b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f\") " Nov 03 20:28:48.692984 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:48.692935 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f-plugins-config-volume\") pod \"b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f\" (UID: \"b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f\") " Nov 03 20:28:48.693298 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:48.693267 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f-plugins-config-volume" pod "b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f" (UID: "b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:28:48.693575 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:48.693561 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:48.694874 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:48.694850 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f-kube-api-access-487pv" pod "b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f" (UID: "b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f"). InnerVolumeSpecName "kube-api-access-487pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:28:48.795190 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:48.795092 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-487pv\" (UniqueName: \"kubernetes.io/projected/b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f-kube-api-access-487pv\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:48.908156 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:48.908122 4918 generic.go:356] "Generic (PLEG): container finished" podID="b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f" containerID="ac9e9de6f4bf9505f9481047a3da48824dd080196c8bd9beb364e0bb2f971dc6" exitCode=0 Nov 03 20:28:48.908364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:48.908205 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt" Nov 03 20:28:48.908364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:48.908230 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt" event={"ID":"b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f","Type":"ContainerDied","Data":"ac9e9de6f4bf9505f9481047a3da48824dd080196c8bd9beb364e0bb2f971dc6"} Nov 03 20:28:48.908364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:48.908285 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt" event={"ID":"b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f","Type":"ContainerDied","Data":"e34aa25bedc73dccca4df66c0c0e25cd3a181058cb0fd85abbb7e7590d67f7b7"} Nov 03 20:28:48.908364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:48.908317 4918 scope.go:122] "RemoveContainer" containerID="ac9e9de6f4bf9505f9481047a3da48824dd080196c8bd9beb364e0bb2f971dc6" Nov 03 20:28:48.920977 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:48.920960 4918 scope.go:122] "RemoveContainer" containerID="ac9e9de6f4bf9505f9481047a3da48824dd080196c8bd9beb364e0bb2f971dc6" Nov 03 20:28:48.921273 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:28:48.921253 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac9e9de6f4bf9505f9481047a3da48824dd080196c8bd9beb364e0bb2f971dc6\": container with ID starting with ac9e9de6f4bf9505f9481047a3da48824dd080196c8bd9beb364e0bb2f971dc6 not found: ID does not exist" containerID="ac9e9de6f4bf9505f9481047a3da48824dd080196c8bd9beb364e0bb2f971dc6" Nov 03 20:28:48.921353 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:48.921281 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac9e9de6f4bf9505f9481047a3da48824dd080196c8bd9beb364e0bb2f971dc6"} err="failed to get container status \"ac9e9de6f4bf9505f9481047a3da48824dd080196c8bd9beb364e0bb2f971dc6\": rpc error: code = NotFound desc = could not find container \"ac9e9de6f4bf9505f9481047a3da48824dd080196c8bd9beb364e0bb2f971dc6\": container with ID starting with ac9e9de6f4bf9505f9481047a3da48824dd080196c8bd9beb364e0bb2f971dc6 not found: ID does not exist" Nov 03 20:28:48.934995 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:48.934970 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt"] Nov 03 20:28:48.939529 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:48.939507 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d9f082-7976878c56-bkmdt"] Nov 03 20:28:49.218083 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:49.218048 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f" path="/var/lib/kubelet/pods/b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f/volumes" Nov 03 20:28:53.505499 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.504294 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m"] Nov 03 20:28:53.505985 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.505758 4918 cpu_manager.go:414] "containerMap: removing container" podUID="b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f" containerName="payload-processing" Nov 03 20:28:53.505985 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.505782 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f" containerName="payload-processing" Nov 03 20:28:53.506116 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.506043 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ac3141-ef2a-4b03-9c26-8cc5cc36a12f" containerName="payload-processing" Nov 03 20:28:53.511030 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.511007 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:53.518956 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.518917 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c60d2a\"/\"default-dockercfg-wnjgm\"" Nov 03 20:28:53.519112 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.519061 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c60d2a\"/\"test-model-c60d2a-kserve-self-signed-certs\"" Nov 03 20:28:53.525164 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.523419 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c60d2a\"/\"kube-root-ca.crt\"" Nov 03 20:28:53.525164 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.523507 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c60d2a\"/\"openshift-service-ca.crt\"" Nov 03 20:28:53.525164 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.525123 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m"] Nov 03 20:28:53.586779 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.586744 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-tls-certs\") pod \"test-model-c60d2a-kserve-65d5f4b98c-pc68m\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:53.586961 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.586790 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-home\") pod \"test-model-c60d2a-kserve-65d5f4b98c-pc68m\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:53.587005 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.586986 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-dshm\") pod \"test-model-c60d2a-kserve-65d5f4b98c-pc68m\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:53.587157 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.587142 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sztw7\" (UniqueName: \"kubernetes.io/projected/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-kube-api-access-sztw7\") pod \"test-model-c60d2a-kserve-65d5f4b98c-pc68m\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:53.587329 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.587315 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-model-cache\") pod \"test-model-c60d2a-kserve-65d5f4b98c-pc68m\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:53.587418 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.587344 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-tmp-dir\") pod \"test-model-c60d2a-kserve-65d5f4b98c-pc68m\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:53.704203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.696298 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-tls-certs\") pod \"test-model-c60d2a-kserve-65d5f4b98c-pc68m\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:53.704203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.696364 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-home\") pod \"test-model-c60d2a-kserve-65d5f4b98c-pc68m\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:53.704203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.696516 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-dshm\") pod \"test-model-c60d2a-kserve-65d5f4b98c-pc68m\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:53.704203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.696757 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sztw7\" (UniqueName: \"kubernetes.io/projected/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-kube-api-access-sztw7\") pod \"test-model-c60d2a-kserve-65d5f4b98c-pc68m\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:53.704203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.696945 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-model-cache\") pod \"test-model-c60d2a-kserve-65d5f4b98c-pc68m\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:53.704203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.696991 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-tmp-dir\") pod \"test-model-c60d2a-kserve-65d5f4b98c-pc68m\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:53.704203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.697759 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-tmp-dir\") pod \"test-model-c60d2a-kserve-65d5f4b98c-pc68m\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:53.708209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.707197 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-model-cache\") pod \"test-model-c60d2a-kserve-65d5f4b98c-pc68m\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:53.731202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.708522 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-home\") pod \"test-model-c60d2a-kserve-65d5f4b98c-pc68m\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:53.735201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.735092 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-tls-certs\") pod \"test-model-c60d2a-kserve-65d5f4b98c-pc68m\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:53.772857 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.764950 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-dshm\") pod \"test-model-c60d2a-kserve-65d5f4b98c-pc68m\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:53.792252 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.790607 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sztw7\" (UniqueName: \"kubernetes.io/projected/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-kube-api-access-sztw7\") pod \"test-model-c60d2a-kserve-65d5f4b98c-pc68m\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:53.846169 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:53.839942 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:54.150368 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:28:54.150330 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4aaf5a9a_91e9_48d8_b69d_97de7d99d51a.slice/crio-627b0a8206bd36cb6c86c73d4de06e7daffd64c494df0217f26e8afd6f3462d1 WatchSource:0}: Error finding container 627b0a8206bd36cb6c86c73d4de06e7daffd64c494df0217f26e8afd6f3462d1: Status 404 returned error can't find the container with id 627b0a8206bd36cb6c86c73d4de06e7daffd64c494df0217f26e8afd6f3462d1 Nov 03 20:28:54.150677 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:54.150656 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m"] Nov 03 20:28:54.990710 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:54.990673 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" event={"ID":"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a","Type":"ContainerStarted","Data":"ddded8e2ea8ab7846d5a6475e41b70207e050debee74c55a0faa1b0ec9f66f73"} Nov 03 20:28:54.991128 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:54.990723 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" event={"ID":"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a","Type":"ContainerStarted","Data":"627b0a8206bd36cb6c86c73d4de06e7daffd64c494df0217f26e8afd6f3462d1"} Nov 03 20:28:55.014946 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:55.014899 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" podStartSLOduration=2.01488488 podStartE2EDuration="2.01488488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:55.01013113 +0000 UTC m=+1917.941998664" watchObservedRunningTime="2026-11-03 20:28:55.01488488 +0000 UTC m=+1917.946752438" Nov 03 20:28:55.840718 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:28:55.840680 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:28:58.211393 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:28:58.211357 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:29:03.841283 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:03.841251 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:29:03.854109 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:03.854076 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:29:04.077458 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:04.077425 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:29:16.168254 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:16.163707 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd"] Nov 03 20:29:16.168254 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:16.164047 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" podUID="0b995af4-64ca-4566-af7f-abd21c9cdb7f" containerName="istio-proxy" containerID="cri-o://c110b7a20ac1757a19ca21526d3af8ac7f5ad63cd9a1a1ccefef00275f571ac7" gracePeriod=30 Nov 03 20:29:17.429297 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:17.429262 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25"] Nov 03 20:29:21.453942 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.453698 4918 generic.go:356] "Generic (PLEG): container finished" podID="0b995af4-64ca-4566-af7f-abd21c9cdb7f" containerID="c110b7a20ac1757a19ca21526d3af8ac7f5ad63cd9a1a1ccefef00275f571ac7" exitCode=0 Nov 03 20:29:21.453942 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.453814 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" event={"ID":"0b995af4-64ca-4566-af7f-abd21c9cdb7f","Type":"ContainerDied","Data":"c110b7a20ac1757a19ca21526d3af8ac7f5ad63cd9a1a1ccefef00275f571ac7"} Nov 03 20:29:21.550261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.550233 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:29:21.645775 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.645725 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-workload-socket\") pod \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " Nov 03 20:29:21.646000 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.645817 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-data\") pod \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " Nov 03 20:29:21.646000 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.645899 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-podinfo\") pod \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " Nov 03 20:29:21.646000 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.645992 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-token\" (UniqueName: \"kubernetes.io/projected/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-token\") pod \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " Nov 03 20:29:21.646207 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.646052 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-workload-socket" pod "0b995af4-64ca-4566-af7f-abd21c9cdb7f" (UID: "0b995af4-64ca-4566-af7f-abd21c9cdb7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.646207 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.646082 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-envoy\") pod \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " Nov 03 20:29:21.646207 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.646151 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b995af4-64ca-4566-af7f-abd21c9cdb7f-kube-api-access-n22kz\" (UniqueName: \"kubernetes.io/projected/0b995af4-64ca-4566-af7f-abd21c9cdb7f-kube-api-access-n22kz\") pod \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " Nov 03 20:29:21.646384 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.646371 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istiod-ca-cert\") pod \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " Nov 03 20:29:21.646439 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.646412 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-workload-certs\") pod \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " Nov 03 20:29:21.646824 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.646581 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-credential-socket\") pod \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\" (UID: \"0b995af4-64ca-4566-af7f-abd21c9cdb7f\") " Nov 03 20:29:21.646945 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.646886 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-data" pod "0b995af4-64ca-4566-af7f-abd21c9cdb7f" (UID: "0b995af4-64ca-4566-af7f-abd21c9cdb7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.647108 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.647088 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-workload-certs" pod "0b995af4-64ca-4566-af7f-abd21c9cdb7f" (UID: "0b995af4-64ca-4566-af7f-abd21c9cdb7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.647492 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.647465 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-credential-socket" pod "0b995af4-64ca-4566-af7f-abd21c9cdb7f" (UID: "0b995af4-64ca-4566-af7f-abd21c9cdb7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.647705 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.647684 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:21.647797 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.647786 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:21.647874 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.647863 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:21.648281 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.648261 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istiod-ca-cert" pod "0b995af4-64ca-4566-af7f-abd21c9cdb7f" (UID: "0b995af4-64ca-4566-af7f-abd21c9cdb7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:21.650477 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.650448 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-podinfo" pod "0b995af4-64ca-4566-af7f-abd21c9cdb7f" (UID: "0b995af4-64ca-4566-af7f-abd21c9cdb7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:29:21.650575 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.650492 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-token" pod "0b995af4-64ca-4566-af7f-abd21c9cdb7f" (UID: "0b995af4-64ca-4566-af7f-abd21c9cdb7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:21.650575 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.650530 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b995af4-64ca-4566-af7f-abd21c9cdb7f-kube-api-access-n22kz" pod "0b995af4-64ca-4566-af7f-abd21c9cdb7f" (UID: "0b995af4-64ca-4566-af7f-abd21c9cdb7f"). InnerVolumeSpecName "kube-api-access-n22kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:21.650839 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.650817 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-envoy" pod "0b995af4-64ca-4566-af7f-abd21c9cdb7f" (UID: "0b995af4-64ca-4566-af7f-abd21c9cdb7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.749496 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.749406 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:21.749496 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.749439 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:21.749496 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.749453 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:21.749496 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.749466 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:21.749496 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.749478 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n22kz\" (UniqueName: \"kubernetes.io/projected/0b995af4-64ca-4566-af7f-abd21c9cdb7f-kube-api-access-n22kz\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:21.749496 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:21.749492 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b995af4-64ca-4566-af7f-abd21c9cdb7f-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:22.429775 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:22.429725 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25" podUID="803e64a2-6d50-49ac-aeef-7c7148e79e82" containerName="payload-processing" containerID="cri-o://b03988dfb85987b23dd3bbec0f85aa0a05acde8f0cf75ca70704405f365bb2a5" gracePeriod=25 Nov 03 20:29:22.463340 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:22.463315 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" Nov 03 20:29:22.463716 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:22.463341 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd" event={"ID":"0b995af4-64ca-4566-af7f-abd21c9cdb7f","Type":"ContainerDied","Data":"7b8784e31dbfba3c35304573b4b48b73d97e2cb0750c4947cf9291f3057ee35c"} Nov 03 20:29:22.463716 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:22.463423 4918 scope.go:122] "RemoveContainer" containerID="c110b7a20ac1757a19ca21526d3af8ac7f5ad63cd9a1a1ccefef00275f571ac7" Nov 03 20:29:22.521317 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:22.521289 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd"] Nov 03 20:29:22.527425 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:22.527399 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wg9nd"] Nov 03 20:29:22.755123 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:22.755091 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25" Nov 03 20:29:22.897050 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:22.897017 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/803e64a2-6d50-49ac-aeef-7c7148e79e82-kube-api-access-zbstm\" (UniqueName: \"kubernetes.io/projected/803e64a2-6d50-49ac-aeef-7c7148e79e82-kube-api-access-zbstm\") pod \"803e64a2-6d50-49ac-aeef-7c7148e79e82\" (UID: \"803e64a2-6d50-49ac-aeef-7c7148e79e82\") " Nov 03 20:29:22.897255 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:22.897104 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/803e64a2-6d50-49ac-aeef-7c7148e79e82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803e64a2-6d50-49ac-aeef-7c7148e79e82-plugins-config-volume\") pod \"803e64a2-6d50-49ac-aeef-7c7148e79e82\" (UID: \"803e64a2-6d50-49ac-aeef-7c7148e79e82\") " Nov 03 20:29:22.897551 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:22.897523 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/803e64a2-6d50-49ac-aeef-7c7148e79e82-plugins-config-volume" pod "803e64a2-6d50-49ac-aeef-7c7148e79e82" (UID: "803e64a2-6d50-49ac-aeef-7c7148e79e82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:22.897985 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:22.897965 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803e64a2-6d50-49ac-aeef-7c7148e79e82-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:22.899298 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:22.899272 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803e64a2-6d50-49ac-aeef-7c7148e79e82-kube-api-access-zbstm" pod "803e64a2-6d50-49ac-aeef-7c7148e79e82" (UID: "803e64a2-6d50-49ac-aeef-7c7148e79e82"). InnerVolumeSpecName "kube-api-access-zbstm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:22.999922 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:22.999832 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbstm\" (UniqueName: \"kubernetes.io/projected/803e64a2-6d50-49ac-aeef-7c7148e79e82-kube-api-access-zbstm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:23.219306 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:23.219266 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b995af4-64ca-4566-af7f-abd21c9cdb7f" path="/var/lib/kubelet/pods/0b995af4-64ca-4566-af7f-abd21c9cdb7f/volumes" Nov 03 20:29:23.473843 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:23.473811 4918 generic.go:356] "Generic (PLEG): container finished" podID="803e64a2-6d50-49ac-aeef-7c7148e79e82" containerID="b03988dfb85987b23dd3bbec0f85aa0a05acde8f0cf75ca70704405f365bb2a5" exitCode=0 Nov 03 20:29:23.474371 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:23.473884 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25" Nov 03 20:29:23.474371 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:23.473892 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25" event={"ID":"803e64a2-6d50-49ac-aeef-7c7148e79e82","Type":"ContainerDied","Data":"b03988dfb85987b23dd3bbec0f85aa0a05acde8f0cf75ca70704405f365bb2a5"} Nov 03 20:29:23.474371 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:23.473945 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25" event={"ID":"803e64a2-6d50-49ac-aeef-7c7148e79e82","Type":"ContainerDied","Data":"6f70a8bc13fd944cc89ae99827218c274b8f2061ef50a05912863c82b65cf8fa"} Nov 03 20:29:23.474371 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:23.473968 4918 scope.go:122] "RemoveContainer" containerID="b03988dfb85987b23dd3bbec0f85aa0a05acde8f0cf75ca70704405f365bb2a5" Nov 03 20:29:23.495642 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:23.495615 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25"] Nov 03 20:29:23.500038 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:23.500008 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7fd1a7-8c8dfdcf8-pzc25"] Nov 03 20:29:23.510596 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:23.510578 4918 scope.go:122] "RemoveContainer" containerID="b03988dfb85987b23dd3bbec0f85aa0a05acde8f0cf75ca70704405f365bb2a5" Nov 03 20:29:23.510928 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:29:23.510906 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03988dfb85987b23dd3bbec0f85aa0a05acde8f0cf75ca70704405f365bb2a5\": container with ID starting with b03988dfb85987b23dd3bbec0f85aa0a05acde8f0cf75ca70704405f365bb2a5 not found: ID does not exist" containerID="b03988dfb85987b23dd3bbec0f85aa0a05acde8f0cf75ca70704405f365bb2a5" Nov 03 20:29:23.510985 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:23.510937 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03988dfb85987b23dd3bbec0f85aa0a05acde8f0cf75ca70704405f365bb2a5"} err="failed to get container status \"b03988dfb85987b23dd3bbec0f85aa0a05acde8f0cf75ca70704405f365bb2a5\": rpc error: code = NotFound desc = could not find container \"b03988dfb85987b23dd3bbec0f85aa0a05acde8f0cf75ca70704405f365bb2a5\": container with ID starting with b03988dfb85987b23dd3bbec0f85aa0a05acde8f0cf75ca70704405f365bb2a5 not found: ID does not exist" Nov 03 20:29:25.218896 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:25.218856 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="803e64a2-6d50-49ac-aeef-7c7148e79e82" path="/var/lib/kubelet/pods/803e64a2-6d50-49ac-aeef-7c7148e79e82/volumes" Nov 03 20:29:28.254725 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:29:28.254689 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:29:38.608852 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.608810 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf"] Nov 03 20:29:38.610085 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.610061 4918 cpu_manager.go:414] "containerMap: removing container" podUID="803e64a2-6d50-49ac-aeef-7c7148e79e82" containerName="payload-processing" Nov 03 20:29:38.610206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.610086 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="803e64a2-6d50-49ac-aeef-7c7148e79e82" containerName="payload-processing" Nov 03 20:29:38.610206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.610100 4918 cpu_manager.go:414] "containerMap: removing container" podUID="0b995af4-64ca-4566-af7f-abd21c9cdb7f" containerName="istio-proxy" Nov 03 20:29:38.610206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.610107 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b995af4-64ca-4566-af7f-abd21c9cdb7f" containerName="istio-proxy" Nov 03 20:29:38.610386 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.610323 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="803e64a2-6d50-49ac-aeef-7c7148e79e82" containerName="payload-processing" Nov 03 20:29:38.610386 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.610336 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b995af4-64ca-4566-af7f-abd21c9cdb7f" containerName="istio-proxy" Nov 03 20:29:38.627730 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.627704 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.632045 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.632015 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-45e569-openshift-default-dockercfg-vqb8n\"" Nov 03 20:29:38.637547 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.637523 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf"] Nov 03 20:29:38.770323 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.770288 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-workload-socket\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.770491 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.770355 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-workload-certs\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.770491 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.770448 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-envoy\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.770566 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.770499 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-podinfo\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.770566 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.770551 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzsxf\" (UniqueName: \"kubernetes.io/projected/22993cbd-6df5-493a-b19a-3f1269dbc57c-kube-api-access-qzsxf\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.770637 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.770625 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-credential-socket\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.770697 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.770684 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-data\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.770825 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.770804 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22993cbd-6df5-493a-b19a-3f1269dbc57c-istiod-ca-cert\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.770962 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.770941 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-token\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.872380 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.872294 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22993cbd-6df5-493a-b19a-3f1269dbc57c-istiod-ca-cert\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.872525 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.872407 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-token\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.872525 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.872481 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-workload-socket\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.872623 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.872539 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-workload-certs\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.872682 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.872618 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-envoy\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.872740 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.872684 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-podinfo\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.872740 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.872721 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzsxf\" (UniqueName: \"kubernetes.io/projected/22993cbd-6df5-493a-b19a-3f1269dbc57c-kube-api-access-qzsxf\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.872851 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.872834 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-credential-socket\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.872908 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.872892 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-data\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.872972 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.872900 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-workload-socket\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.872972 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.872948 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-workload-certs\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.873157 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.873133 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22993cbd-6df5-493a-b19a-3f1269dbc57c-istiod-ca-cert\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.873277 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.873234 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-data\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.873326 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.873308 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-credential-socket\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.874953 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.874929 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-envoy\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.875231 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.875208 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-podinfo\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.886734 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.886701 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-token\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.886857 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.886837 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzsxf\" (UniqueName: \"kubernetes.io/projected/22993cbd-6df5-493a-b19a-3f1269dbc57c-kube-api-access-qzsxf\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:38.950203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:38.949940 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:39.290258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:39.290226 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf"] Nov 03 20:29:39.290588 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:29:39.290551 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22993cbd_6df5_493a_b19a_3f1269dbc57c.slice/crio-271b1622425b4ced6863d6583ecc6de4b8917258929a56a35ca99f5c670599df WatchSource:0}: Error finding container 271b1622425b4ced6863d6583ecc6de4b8917258929a56a35ca99f5c670599df: Status 404 returned error can't find the container with id 271b1622425b4ced6863d6583ecc6de4b8917258929a56a35ca99f5c670599df Nov 03 20:29:39.293633 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:39.293593 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:29:39.293748 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:39.293704 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:29:39.293815 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:39.293751 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:29:39.719041 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:39.718998 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" event={"ID":"22993cbd-6df5-493a-b19a-3f1269dbc57c","Type":"ContainerStarted","Data":"37ddbaa9ecad3ef24ff76a2ea1b88848a94ccd7c198e440a96376cb5950e6c8e"} Nov 03 20:29:39.719443 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:39.719070 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" event={"ID":"22993cbd-6df5-493a-b19a-3f1269dbc57c","Type":"ContainerStarted","Data":"271b1622425b4ced6863d6583ecc6de4b8917258929a56a35ca99f5c670599df"} Nov 03 20:29:39.746838 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:39.746769 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" podStartSLOduration=1.7467519280000001 podStartE2EDuration="1.746751928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:39.742832369 +0000 UTC m=+1962.674699880" watchObservedRunningTime="2026-11-03 20:29:39.746751928 +0000 UTC m=+1962.678619479" Nov 03 20:29:39.950322 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:39.950280 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:40.952770 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:40.952733 4918 patch_prober.go:29] interesting pod/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 03 20:29:40.953277 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:40.952792 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" podUID="22993cbd-6df5-493a-b19a-3f1269dbc57c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 03 20:29:41.949913 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:41.949873 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz"] Nov 03 20:29:41.953087 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:41.952501 4918 patch_prober.go:29] interesting pod/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 03 20:29:41.953726 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:41.953084 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" podUID="22993cbd-6df5-493a-b19a-3f1269dbc57c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 03 20:29:41.957537 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:41.957512 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:41.969775 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:41.964903 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8c684d-openshift-default-dockercfg-vqwnd\"" Nov 03 20:29:41.969775 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:41.968508 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz"] Nov 03 20:29:42.194921 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.194883 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-podinfo\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.195130 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.195105 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istiod-ca-cert\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.195283 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.195263 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-workload-socket\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.195410 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.195382 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-credential-socket\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.195513 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.195498 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-envoy\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.195577 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.195563 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-workload-certs\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.195636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.195624 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-token\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.195692 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.195666 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6mmj\" (UniqueName: \"kubernetes.io/projected/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-kube-api-access-f6mmj\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.195779 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.195758 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-data\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.297550 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.297452 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istiod-ca-cert\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.297550 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.297546 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-workload-socket\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.297794 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.297627 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-credential-socket\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.297794 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.297723 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-envoy\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.297794 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.297775 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-workload-certs\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.297955 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.297826 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-token\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.297955 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.297857 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6mmj\" (UniqueName: \"kubernetes.io/projected/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-kube-api-access-f6mmj\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.297955 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.297923 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-data\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.298122 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.297987 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-workload-socket\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.298122 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.298029 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-podinfo\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.298122 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.298037 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-credential-socket\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.298338 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.298276 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istiod-ca-cert\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.298456 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.298434 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-workload-certs\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.298517 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.298468 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-data\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.300296 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.300271 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-envoy\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.300671 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.300650 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-podinfo\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.310296 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.310275 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-token\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.310584 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.310561 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6mmj\" (UniqueName: \"kubernetes.io/projected/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-kube-api-access-f6mmj\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.599818 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.599783 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:42.961911 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.961864 4918 patch_prober.go:29] interesting pod/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 03 20:29:42.962376 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.961940 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" podUID="22993cbd-6df5-493a-b19a-3f1269dbc57c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 03 20:29:42.970801 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.970738 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz"] Nov 03 20:29:42.981620 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.981586 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:29:42.981816 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.981795 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:29:42.981933 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:42.981918 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:29:43.780116 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:43.780074 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" event={"ID":"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5","Type":"ContainerStarted","Data":"65bfdb998892e61eafa3dcc00a87bffc1735424b8d82976a086b8e4bd5b860aa"} Nov 03 20:29:43.780353 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:43.780164 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" event={"ID":"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5","Type":"ContainerStarted","Data":"735fb35d9dd9334ec06ca4579feaa69efc834eb531fa509413651cffaec7a222"} Nov 03 20:29:43.806892 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:43.806826 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" podStartSLOduration=2.806802491 podStartE2EDuration="2.806802491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:43.805892383 +0000 UTC m=+1966.737759893" watchObservedRunningTime="2026-11-03 20:29:43.806802491 +0000 UTC m=+1966.738670004" Nov 03 20:29:43.975330 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:43.974454 4918 patch_prober.go:29] interesting pod/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 03 20:29:43.975330 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:43.974519 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" podUID="22993cbd-6df5-493a-b19a-3f1269dbc57c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 03 20:29:44.600518 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:44.600484 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:44.602467 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:44.602437 4918 patch_prober.go:29] interesting pod/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 20:29:44.602573 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:44.602493 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" podUID="eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 20:29:44.962262 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:44.961950 4918 patch_prober.go:29] interesting pod/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 03 20:29:44.962262 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:44.962017 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" podUID="22993cbd-6df5-493a-b19a-3f1269dbc57c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 03 20:29:45.600248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:45.600209 4918 patch_prober.go:29] interesting pod/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 20:29:45.600642 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:45.600269 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" podUID="eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 20:29:45.951092 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:45.950995 4918 patch_prober.go:29] interesting pod/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 03 20:29:45.951092 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:45.951060 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" podUID="22993cbd-6df5-493a-b19a-3f1269dbc57c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 03 20:29:46.609199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:46.607285 4918 patch_prober.go:29] interesting pod/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 20:29:46.609199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:46.607347 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" podUID="eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 20:29:46.978309 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:46.975986 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:46.978309 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:46.977708 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:46.981493 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:46.981462 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:29:47.600591 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:47.600552 4918 patch_prober.go:29] interesting pod/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 20:29:47.600815 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:47.600605 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" podUID="eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 20:29:48.601120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:48.601082 4918 patch_prober.go:29] interesting pod/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 20:29:48.601560 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:48.601141 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" podUID="eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 20:29:48.856397 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:48.854493 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n"] Nov 03 20:29:48.856397 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:48.854775 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" podUID="477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" containerName="istio-proxy" containerID="cri-o://baad3552eded59fd592ea97eef9aa2baa2dedf2816289d8cbd996008e5585999" gracePeriod=30 Nov 03 20:29:49.605239 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:49.604683 4918 patch_prober.go:29] interesting pod/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 20:29:49.605239 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:49.604802 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" podUID="eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 20:29:50.607496 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:50.607463 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:50.607978 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:50.607879 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:50.608532 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:50.608501 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:29:54.075067 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.075034 4918 generic.go:356] "Generic (PLEG): container finished" podID="477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" containerID="baad3552eded59fd592ea97eef9aa2baa2dedf2816289d8cbd996008e5585999" exitCode=0 Nov 03 20:29:54.075505 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.075196 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" event={"ID":"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944","Type":"ContainerDied","Data":"baad3552eded59fd592ea97eef9aa2baa2dedf2816289d8cbd996008e5585999"} Nov 03 20:29:54.166076 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.166011 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:29:54.206002 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.205970 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istiod-ca-cert\") pod \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " Nov 03 20:29:54.206193 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.206032 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-envoy\") pod \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " Nov 03 20:29:54.206193 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.206075 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-token\" (UniqueName: \"kubernetes.io/projected/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-token\") pod \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " Nov 03 20:29:54.206193 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.206098 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-podinfo\") pod \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " Nov 03 20:29:54.206193 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.206134 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-workload-socket\") pod \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " Nov 03 20:29:54.206448 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.206306 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-workload-certs\") pod \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " Nov 03 20:29:54.206448 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.206344 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-credential-socket\") pod \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " Nov 03 20:29:54.206448 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.206378 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-kube-api-access-dzjmc\" (UniqueName: \"kubernetes.io/projected/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-kube-api-access-dzjmc\") pod \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " Nov 03 20:29:54.206600 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.206436 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istiod-ca-cert" pod "477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" (UID: "477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:54.206663 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.206595 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-data\") pod \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\" (UID: \"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944\") " Nov 03 20:29:54.206792 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.206769 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-workload-certs" pod "477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" (UID: "477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:54.208422 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.206993 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-credential-socket" pod "477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" (UID: "477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:54.208422 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.207118 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-workload-socket" pod "477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" (UID: "477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:54.208422 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.207168 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-data" pod "477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" (UID: "477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:54.208422 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.207698 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:54.208422 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.207722 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:54.208422 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.207737 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:54.208422 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.207752 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:54.208422 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.207768 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:54.209063 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.209030 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-podinfo" pod "477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" (UID: "477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:29:54.209063 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.209042 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-envoy" pod "477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" (UID: "477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:54.209274 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.209079 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-token" pod "477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" (UID: "477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:54.209274 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.209101 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-kube-api-access-dzjmc" pod "477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" (UID: "477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944"). InnerVolumeSpecName "kube-api-access-dzjmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:54.310049 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.310013 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:54.310270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.310057 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:54.310270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.310072 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:54.310270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:54.310086 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzjmc\" (UniqueName: \"kubernetes.io/projected/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944-kube-api-access-dzjmc\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:55.100941 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:55.100907 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" Nov 03 20:29:55.100941 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:55.100917 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n" event={"ID":"477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944","Type":"ContainerDied","Data":"e25d55c19439b1fe5b57196d560aded4c4b167e57e7056567d7ef62fcb5eb7f9"} Nov 03 20:29:55.101477 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:55.101010 4918 scope.go:122] "RemoveContainer" containerID="baad3552eded59fd592ea97eef9aa2baa2dedf2816289d8cbd996008e5585999" Nov 03 20:29:55.132643 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:55.132597 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n"] Nov 03 20:29:55.137908 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:55.137878 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-pkq9n"] Nov 03 20:29:55.220144 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:29:55.220108 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" path="/var/lib/kubelet/pods/477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944/volumes" Nov 03 20:29:58.209408 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:29:58.209362 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:30:00.177695 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.177655 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw"] Nov 03 20:30:00.178895 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.178865 4918 cpu_manager.go:414] "containerMap: removing container" podUID="477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" containerName="istio-proxy" Nov 03 20:30:00.178895 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.178890 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" containerName="istio-proxy" Nov 03 20:30:00.179214 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.179173 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="477a6f6c-a2f6-4aa2-b9ed-f323e5bfc944" containerName="istio-proxy" Nov 03 20:30:00.185318 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.185292 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z"] Nov 03 20:30:00.185451 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.185405 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw" Nov 03 20:30:00.190228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.190196 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z" Nov 03 20:30:00.197854 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.197834 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:30:00.211274 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.211245 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw"] Nov 03 20:30:00.221278 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.221241 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z"] Nov 03 20:30:00.254854 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.254805 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cffzr\" (UniqueName: \"kubernetes.io/projected/cc721d0a-faab-45c5-8476-9aec206a65c3-kube-api-access-cffzr\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z\" (UID: \"cc721d0a-faab-45c5-8476-9aec206a65c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z" Nov 03 20:30:00.255023 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.255006 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wbp5\" (UniqueName: \"kubernetes.io/projected/0832a6d3-76cd-4eb6-9879-c880f98a786f-kube-api-access-6wbp5\") pod \"maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw\" (UID: \"0832a6d3-76cd-4eb6-9879-c880f98a786f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw" Nov 03 20:30:00.255114 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.255098 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0832a6d3-76cd-4eb6-9879-c880f98a786f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw\" (UID: \"0832a6d3-76cd-4eb6-9879-c880f98a786f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw" Nov 03 20:30:00.255160 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.255132 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc721d0a-faab-45c5-8476-9aec206a65c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z\" (UID: \"cc721d0a-faab-45c5-8476-9aec206a65c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z" Nov 03 20:30:00.256720 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.256692 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t"] Nov 03 20:30:00.261876 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.261849 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" Nov 03 20:30:00.264437 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.264410 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2"] Nov 03 20:30:00.269818 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.269789 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" Nov 03 20:30:00.325914 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.325875 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx"] Nov 03 20:30:00.332934 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.332261 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t"] Nov 03 20:30:00.332934 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.332314 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" Nov 03 20:30:00.362143 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.361266 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0832a6d3-76cd-4eb6-9879-c880f98a786f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw\" (UID: \"0832a6d3-76cd-4eb6-9879-c880f98a786f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw" Nov 03 20:30:00.362143 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.361344 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc721d0a-faab-45c5-8476-9aec206a65c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z\" (UID: \"cc721d0a-faab-45c5-8476-9aec206a65c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z" Nov 03 20:30:00.365228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.365147 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/752114fb-ce1f-4ddb-8a69-75c8a44a364f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2\" (UID: \"752114fb-ce1f-4ddb-8a69-75c8a44a364f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" Nov 03 20:30:00.365502 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.365480 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cffzr\" (UniqueName: \"kubernetes.io/projected/cc721d0a-faab-45c5-8476-9aec206a65c3-kube-api-access-cffzr\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z\" (UID: \"cc721d0a-faab-45c5-8476-9aec206a65c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z" Nov 03 20:30:00.365578 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.365530 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxft2\" (UniqueName: \"kubernetes.io/projected/7a2653fa-4e55-48c8-a756-1f24c4c8412a-kube-api-access-mxft2\") pod \"maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t\" (UID: \"7a2653fa-4e55-48c8-a756-1f24c4c8412a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" Nov 03 20:30:00.367104 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.367058 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghf9k\" (UniqueName: \"kubernetes.io/projected/752114fb-ce1f-4ddb-8a69-75c8a44a364f-kube-api-access-ghf9k\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2\" (UID: \"752114fb-ce1f-4ddb-8a69-75c8a44a364f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" Nov 03 20:30:00.367298 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.367136 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wbp5\" (UniqueName: \"kubernetes.io/projected/0832a6d3-76cd-4eb6-9879-c880f98a786f-kube-api-access-6wbp5\") pod \"maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw\" (UID: \"0832a6d3-76cd-4eb6-9879-c880f98a786f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw" Nov 03 20:30:00.367298 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.367194 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a2653fa-4e55-48c8-a756-1f24c4c8412a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t\" (UID: \"7a2653fa-4e55-48c8-a756-1f24c4c8412a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" Nov 03 20:30:00.368991 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.368089 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0832a6d3-76cd-4eb6-9879-c880f98a786f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw\" (UID: \"0832a6d3-76cd-4eb6-9879-c880f98a786f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw" Nov 03 20:30:00.368991 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.368904 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc721d0a-faab-45c5-8476-9aec206a65c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z\" (UID: \"cc721d0a-faab-45c5-8476-9aec206a65c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z" Nov 03 20:30:00.380062 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.379664 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx"] Nov 03 20:30:00.389248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.388921 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2"] Nov 03 20:30:00.407528 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.407497 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92"] Nov 03 20:30:00.417592 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.412045 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wbp5\" (UniqueName: \"kubernetes.io/projected/0832a6d3-76cd-4eb6-9879-c880f98a786f-kube-api-access-6wbp5\") pod \"maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw\" (UID: \"0832a6d3-76cd-4eb6-9879-c880f98a786f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw" Nov 03 20:30:00.417592 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.414015 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" Nov 03 20:30:00.417592 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.414206 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8"] Nov 03 20:30:00.419491 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.419470 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" Nov 03 20:30:00.427507 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.427484 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cffzr\" (UniqueName: \"kubernetes.io/projected/cc721d0a-faab-45c5-8476-9aec206a65c3-kube-api-access-cffzr\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z\" (UID: \"cc721d0a-faab-45c5-8476-9aec206a65c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z" Nov 03 20:30:00.469297 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.469219 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghf9k\" (UniqueName: \"kubernetes.io/projected/752114fb-ce1f-4ddb-8a69-75c8a44a364f-kube-api-access-ghf9k\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2\" (UID: \"752114fb-ce1f-4ddb-8a69-75c8a44a364f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" Nov 03 20:30:00.469297 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.469280 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a2653fa-4e55-48c8-a756-1f24c4c8412a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t\" (UID: \"7a2653fa-4e55-48c8-a756-1f24c4c8412a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" Nov 03 20:30:00.469534 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.469294 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92"] Nov 03 20:30:00.469534 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.469329 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81a01d22-8738-4955-9dc5-73d769266ddc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx\" (UID: \"81a01d22-8738-4955-9dc5-73d769266ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" Nov 03 20:30:00.469534 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.469416 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6st28\" (UniqueName: \"kubernetes.io/projected/c4cccb65-6424-4e0b-85bf-6f1fba8d5572-kube-api-access-6st28\") pod \"maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8\" (UID: \"c4cccb65-6424-4e0b-85bf-6f1fba8d5572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" Nov 03 20:30:00.469666 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.469562 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/752114fb-ce1f-4ddb-8a69-75c8a44a364f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2\" (UID: \"752114fb-ce1f-4ddb-8a69-75c8a44a364f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" Nov 03 20:30:00.469666 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.469601 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2228ed1d-945a-4b33-ba98-ee8b52cbf902-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92\" (UID: \"2228ed1d-945a-4b33-ba98-ee8b52cbf902\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" Nov 03 20:30:00.469666 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.469637 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4cccb65-6424-4e0b-85bf-6f1fba8d5572-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8\" (UID: \"c4cccb65-6424-4e0b-85bf-6f1fba8d5572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" Nov 03 20:30:00.469783 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.469680 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kg8h\" (UniqueName: \"kubernetes.io/projected/2228ed1d-945a-4b33-ba98-ee8b52cbf902-kube-api-access-6kg8h\") pod \"maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92\" (UID: \"2228ed1d-945a-4b33-ba98-ee8b52cbf902\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" Nov 03 20:30:00.469783 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.469737 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxft2\" (UniqueName: \"kubernetes.io/projected/7a2653fa-4e55-48c8-a756-1f24c4c8412a-kube-api-access-mxft2\") pod \"maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t\" (UID: \"7a2653fa-4e55-48c8-a756-1f24c4c8412a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" Nov 03 20:30:00.469925 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.469853 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64kct\" (UniqueName: \"kubernetes.io/projected/81a01d22-8738-4955-9dc5-73d769266ddc-kube-api-access-64kct\") pod \"maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx\" (UID: \"81a01d22-8738-4955-9dc5-73d769266ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" Nov 03 20:30:00.470101 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.470075 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a2653fa-4e55-48c8-a756-1f24c4c8412a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t\" (UID: \"7a2653fa-4e55-48c8-a756-1f24c4c8412a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" Nov 03 20:30:00.470453 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.470429 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/752114fb-ce1f-4ddb-8a69-75c8a44a364f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2\" (UID: \"752114fb-ce1f-4ddb-8a69-75c8a44a364f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" Nov 03 20:30:00.484699 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.484640 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8"] Nov 03 20:30:00.503232 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.503166 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghf9k\" (UniqueName: \"kubernetes.io/projected/752114fb-ce1f-4ddb-8a69-75c8a44a364f-kube-api-access-ghf9k\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2\" (UID: \"752114fb-ce1f-4ddb-8a69-75c8a44a364f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" Nov 03 20:30:00.503671 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.503611 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxft2\" (UniqueName: \"kubernetes.io/projected/7a2653fa-4e55-48c8-a756-1f24c4c8412a-kube-api-access-mxft2\") pod \"maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t\" (UID: \"7a2653fa-4e55-48c8-a756-1f24c4c8412a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" Nov 03 20:30:00.511120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.511092 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw" Nov 03 20:30:00.537218 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.537193 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z" Nov 03 20:30:00.572016 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.571976 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4cccb65-6424-4e0b-85bf-6f1fba8d5572-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8\" (UID: \"c4cccb65-6424-4e0b-85bf-6f1fba8d5572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" Nov 03 20:30:00.572143 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.572067 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kg8h\" (UniqueName: \"kubernetes.io/projected/2228ed1d-945a-4b33-ba98-ee8b52cbf902-kube-api-access-6kg8h\") pod \"maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92\" (UID: \"2228ed1d-945a-4b33-ba98-ee8b52cbf902\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" Nov 03 20:30:00.572257 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.572237 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64kct\" (UniqueName: \"kubernetes.io/projected/81a01d22-8738-4955-9dc5-73d769266ddc-kube-api-access-64kct\") pod \"maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx\" (UID: \"81a01d22-8738-4955-9dc5-73d769266ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" Nov 03 20:30:00.572386 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.572371 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81a01d22-8738-4955-9dc5-73d769266ddc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx\" (UID: \"81a01d22-8738-4955-9dc5-73d769266ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" Nov 03 20:30:00.572446 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.572432 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6st28\" (UniqueName: \"kubernetes.io/projected/c4cccb65-6424-4e0b-85bf-6f1fba8d5572-kube-api-access-6st28\") pod \"maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8\" (UID: \"c4cccb65-6424-4e0b-85bf-6f1fba8d5572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" Nov 03 20:30:00.572663 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.572647 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2228ed1d-945a-4b33-ba98-ee8b52cbf902-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92\" (UID: \"2228ed1d-945a-4b33-ba98-ee8b52cbf902\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" Nov 03 20:30:00.573634 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.573522 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2228ed1d-945a-4b33-ba98-ee8b52cbf902-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92\" (UID: \"2228ed1d-945a-4b33-ba98-ee8b52cbf902\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" Nov 03 20:30:00.574233 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.574160 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4cccb65-6424-4e0b-85bf-6f1fba8d5572-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8\" (UID: \"c4cccb65-6424-4e0b-85bf-6f1fba8d5572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" Nov 03 20:30:00.577503 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.575391 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81a01d22-8738-4955-9dc5-73d769266ddc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx\" (UID: \"81a01d22-8738-4955-9dc5-73d769266ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" Nov 03 20:30:00.583461 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.583439 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" Nov 03 20:30:00.606859 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.599230 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" Nov 03 20:30:00.657203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.657113 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6st28\" (UniqueName: \"kubernetes.io/projected/c4cccb65-6424-4e0b-85bf-6f1fba8d5572-kube-api-access-6st28\") pod \"maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8\" (UID: \"c4cccb65-6424-4e0b-85bf-6f1fba8d5572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" Nov 03 20:30:00.658382 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.658022 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kg8h\" (UniqueName: \"kubernetes.io/projected/2228ed1d-945a-4b33-ba98-ee8b52cbf902-kube-api-access-6kg8h\") pod \"maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92\" (UID: \"2228ed1d-945a-4b33-ba98-ee8b52cbf902\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" Nov 03 20:30:00.659751 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.659339 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64kct\" (UniqueName: \"kubernetes.io/projected/81a01d22-8738-4955-9dc5-73d769266ddc-kube-api-access-64kct\") pod \"maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx\" (UID: \"81a01d22-8738-4955-9dc5-73d769266ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" Nov 03 20:30:00.796816 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.794499 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" Nov 03 20:30:00.796816 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.794550 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" Nov 03 20:30:00.826692 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.826642 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw"] Nov 03 20:30:00.950356 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.949451 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z"] Nov 03 20:30:00.950504 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:00.949736 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" Nov 03 20:30:01.064981 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:01.064730 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t"] Nov 03 20:30:01.119490 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:01.119449 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92"] Nov 03 20:30:01.151634 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:01.151605 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8"] Nov 03 20:30:01.176495 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:01.176451 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" event={"ID":"7a2653fa-4e55-48c8-a756-1f24c4c8412a","Type":"ContainerStarted","Data":"75236c2145aea45de3fa5ef9b1f9524847a2ef90b177ea5caf9ddaad4ce85071"} Nov 03 20:30:01.177957 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:01.177926 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw" event={"ID":"0832a6d3-76cd-4eb6-9879-c880f98a786f","Type":"ContainerStarted","Data":"fb2ce4c6baebc0075dfa4eb64fb0d88aa88e0b6497cd6a3f2f8faf494eb29265"} Nov 03 20:30:01.179845 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:01.179798 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z" event={"ID":"cc721d0a-faab-45c5-8476-9aec206a65c3","Type":"ContainerStarted","Data":"d8e2f0b0ce9b8a01050e6eb3337bff13b53641884a3f6f6e9cc37b0ebb475fc7"} Nov 03 20:30:01.181593 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:01.181563 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" event={"ID":"2228ed1d-945a-4b33-ba98-ee8b52cbf902","Type":"ContainerStarted","Data":"0f52b89a9b479f0f4da6a5b3eeda23c8407ddd76aae89e1712ab34687d5cb5bb"} Nov 03 20:30:01.222813 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:01.222782 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2"] Nov 03 20:30:01.223644 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:30:01.223619 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4cccb65_6424_4e0b_85bf_6f1fba8d5572.slice/crio-31f7e4ac11f13b6aa6271a054764a772cd22402723971acdb386c34208f45b95 WatchSource:0}: Error finding container 31f7e4ac11f13b6aa6271a054764a772cd22402723971acdb386c34208f45b95: Status 404 returned error can't find the container with id 31f7e4ac11f13b6aa6271a054764a772cd22402723971acdb386c34208f45b95 Nov 03 20:30:01.224819 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:30:01.224798 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752114fb_ce1f_4ddb_8a69_75c8a44a364f.slice/crio-b3b702d9b58cdf26b12264af31b713a626a2606dfd19179ba2569f8afbce5693 WatchSource:0}: Error finding container b3b702d9b58cdf26b12264af31b713a626a2606dfd19179ba2569f8afbce5693: Status 404 returned error can't find the container with id b3b702d9b58cdf26b12264af31b713a626a2606dfd19179ba2569f8afbce5693 Nov 03 20:30:01.279704 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:30:01.279671 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a01d22_8738_4955_9dc5_73d769266ddc.slice/crio-6696f9fc96312b0f37156a8732ca0f051fc80158a1a61d53d16171121bdcc0db WatchSource:0}: Error finding container 6696f9fc96312b0f37156a8732ca0f051fc80158a1a61d53d16171121bdcc0db: Status 404 returned error can't find the container with id 6696f9fc96312b0f37156a8732ca0f051fc80158a1a61d53d16171121bdcc0db Nov 03 20:30:01.280162 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:01.280140 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx"] Nov 03 20:30:02.193460 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:02.193419 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" event={"ID":"81a01d22-8738-4955-9dc5-73d769266ddc","Type":"ContainerStarted","Data":"b2251f29f239024fa30339275c141ba453d8cbcc30c7a56ce4959f9b0aa2c863"} Nov 03 20:30:02.193870 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:02.193490 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" event={"ID":"81a01d22-8738-4955-9dc5-73d769266ddc","Type":"ContainerStarted","Data":"6696f9fc96312b0f37156a8732ca0f051fc80158a1a61d53d16171121bdcc0db"} Nov 03 20:30:02.195276 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:02.195235 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" event={"ID":"c4cccb65-6424-4e0b-85bf-6f1fba8d5572","Type":"ContainerStarted","Data":"9b45eee7ebf9033090c2c04672113df13122dad71bd39393cfa5de3f187213cd"} Nov 03 20:30:02.195380 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:02.195301 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" event={"ID":"c4cccb65-6424-4e0b-85bf-6f1fba8d5572","Type":"ContainerStarted","Data":"31f7e4ac11f13b6aa6271a054764a772cd22402723971acdb386c34208f45b95"} Nov 03 20:30:02.203235 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:02.203206 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" event={"ID":"2228ed1d-945a-4b33-ba98-ee8b52cbf902","Type":"ContainerStarted","Data":"010427b1b3dd1f09b8a070f250946f704ba293504d6cc776f7647d0a49f227ed"} Nov 03 20:30:02.204960 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:02.204932 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" event={"ID":"7a2653fa-4e55-48c8-a756-1f24c4c8412a","Type":"ContainerStarted","Data":"2e88f28840110982deff2203ca9c230cf731dae4eac83c02caa76002bc3d9e00"} Nov 03 20:30:02.206472 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:02.206449 4918 generic.go:356] "Generic (PLEG): container finished" podID="0832a6d3-76cd-4eb6-9879-c880f98a786f" containerID="b75bab864f6bf2be38b6fc48517c3762ec2ca516b9a34a588f5c27a45e799e63" exitCode=60 Nov 03 20:30:02.206567 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:02.206529 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw" event={"ID":"0832a6d3-76cd-4eb6-9879-c880f98a786f","Type":"ContainerDied","Data":"b75bab864f6bf2be38b6fc48517c3762ec2ca516b9a34a588f5c27a45e799e63"} Nov 03 20:30:02.206733 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:02.206715 4918 scope.go:122] "RemoveContainer" containerID="b75bab864f6bf2be38b6fc48517c3762ec2ca516b9a34a588f5c27a45e799e63" Nov 03 20:30:02.208365 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:02.208341 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" event={"ID":"752114fb-ce1f-4ddb-8a69-75c8a44a364f","Type":"ContainerStarted","Data":"26ed325146a7ebe756accba282666ab8d05159ba46b0ada03767d56c270fd59e"} Nov 03 20:30:02.208454 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:02.208392 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" event={"ID":"752114fb-ce1f-4ddb-8a69-75c8a44a364f","Type":"ContainerStarted","Data":"b3b702d9b58cdf26b12264af31b713a626a2606dfd19179ba2569f8afbce5693"} Nov 03 20:30:02.210485 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:02.210350 4918 generic.go:356] "Generic (PLEG): container finished" podID="cc721d0a-faab-45c5-8476-9aec206a65c3" containerID="65ccf1e3f9723476cb5f6bdfc4bfa17e7db1479be50edb0f0984535733b9c9a3" exitCode=60 Nov 03 20:30:02.210485 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:02.210389 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z" event={"ID":"cc721d0a-faab-45c5-8476-9aec206a65c3","Type":"ContainerDied","Data":"65ccf1e3f9723476cb5f6bdfc4bfa17e7db1479be50edb0f0984535733b9c9a3"} Nov 03 20:30:02.210658 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:02.210641 4918 scope.go:122] "RemoveContainer" containerID="65ccf1e3f9723476cb5f6bdfc4bfa17e7db1479be50edb0f0984535733b9c9a3" Nov 03 20:30:02.219299 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:02.219237 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" podStartSLOduration=2.219222516 podStartE2EDuration="2.219222516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.21444607 +0000 UTC m=+1985.146313593" watchObservedRunningTime="2026-11-03 20:30:02.219222516 +0000 UTC m=+1985.151090027" Nov 03 20:30:02.265842 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:02.265785 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" podStartSLOduration=2.265766318 podStartE2EDuration="2.265766318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.261748681 +0000 UTC m=+1985.193616195" watchObservedRunningTime="2026-11-03 20:30:02.265766318 +0000 UTC m=+1985.197633830" Nov 03 20:30:02.287228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:02.287163 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" podStartSLOduration=2.287149269 podStartE2EDuration="2.287149269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.283717139 +0000 UTC m=+1985.215584651" watchObservedRunningTime="2026-11-03 20:30:02.287149269 +0000 UTC m=+1985.219016779" Nov 03 20:30:02.302587 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:02.302542 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" podStartSLOduration=2.302527583 podStartE2EDuration="2.302527583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.301335474 +0000 UTC m=+1985.233202986" watchObservedRunningTime="2026-11-03 20:30:02.302527583 +0000 UTC m=+1985.234395093" Nov 03 20:30:02.323225 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:02.323152 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" podStartSLOduration=2.323133465 podStartE2EDuration="2.323133465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.318936977 +0000 UTC m=+1985.250804490" watchObservedRunningTime="2026-11-03 20:30:02.323133465 +0000 UTC m=+1985.255000978" Nov 03 20:30:02.415585 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:02.415507 4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2653fa_4e55_48c8_a756_1f24c4c8412a.slice/crio-2e88f28840110982deff2203ca9c230cf731dae4eac83c02caa76002bc3d9e00.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2653fa_4e55_48c8_a756_1f24c4c8412a.slice/crio-conmon-2e88f28840110982deff2203ca9c230cf731dae4eac83c02caa76002bc3d9e00.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:30:03.223423 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:03.223390 4918 generic.go:356] "Generic (PLEG): container finished" podID="c4cccb65-6424-4e0b-85bf-6f1fba8d5572" containerID="9b45eee7ebf9033090c2c04672113df13122dad71bd39393cfa5de3f187213cd" exitCode=60 Nov 03 20:30:03.223916 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:03.223477 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" event={"ID":"c4cccb65-6424-4e0b-85bf-6f1fba8d5572","Type":"ContainerDied","Data":"9b45eee7ebf9033090c2c04672113df13122dad71bd39393cfa5de3f187213cd"} Nov 03 20:30:03.223916 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:03.223844 4918 scope.go:122] "RemoveContainer" containerID="9b45eee7ebf9033090c2c04672113df13122dad71bd39393cfa5de3f187213cd" Nov 03 20:30:03.225761 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:03.225726 4918 generic.go:356] "Generic (PLEG): container finished" podID="2228ed1d-945a-4b33-ba98-ee8b52cbf902" containerID="010427b1b3dd1f09b8a070f250946f704ba293504d6cc776f7647d0a49f227ed" exitCode=60 Nov 03 20:30:03.225982 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:03.225863 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" event={"ID":"2228ed1d-945a-4b33-ba98-ee8b52cbf902","Type":"ContainerDied","Data":"010427b1b3dd1f09b8a070f250946f704ba293504d6cc776f7647d0a49f227ed"} Nov 03 20:30:03.226617 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:03.226160 4918 scope.go:122] "RemoveContainer" containerID="010427b1b3dd1f09b8a070f250946f704ba293504d6cc776f7647d0a49f227ed" Nov 03 20:30:03.240732 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:03.229852 4918 generic.go:356] "Generic (PLEG): container finished" podID="7a2653fa-4e55-48c8-a756-1f24c4c8412a" containerID="2e88f28840110982deff2203ca9c230cf731dae4eac83c02caa76002bc3d9e00" exitCode=60 Nov 03 20:30:03.240732 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:03.230004 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" event={"ID":"7a2653fa-4e55-48c8-a756-1f24c4c8412a","Type":"ContainerDied","Data":"2e88f28840110982deff2203ca9c230cf731dae4eac83c02caa76002bc3d9e00"} Nov 03 20:30:03.240732 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:03.230460 4918 scope.go:122] "RemoveContainer" containerID="2e88f28840110982deff2203ca9c230cf731dae4eac83c02caa76002bc3d9e00" Nov 03 20:30:03.246230 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:03.246198 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw" event={"ID":"0832a6d3-76cd-4eb6-9879-c880f98a786f","Type":"ContainerStarted","Data":"c1faf9f51c7614053293b2572594078b34d9845179d416611441aa09f7aaff77"} Nov 03 20:30:03.249233 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:03.249016 4918 generic.go:356] "Generic (PLEG): container finished" podID="752114fb-ce1f-4ddb-8a69-75c8a44a364f" containerID="26ed325146a7ebe756accba282666ab8d05159ba46b0ada03767d56c270fd59e" exitCode=60 Nov 03 20:30:03.249233 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:03.249165 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" event={"ID":"752114fb-ce1f-4ddb-8a69-75c8a44a364f","Type":"ContainerDied","Data":"26ed325146a7ebe756accba282666ab8d05159ba46b0ada03767d56c270fd59e"} Nov 03 20:30:03.249630 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:03.249607 4918 scope.go:122] "RemoveContainer" containerID="26ed325146a7ebe756accba282666ab8d05159ba46b0ada03767d56c270fd59e" Nov 03 20:30:03.252237 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:03.252172 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z" event={"ID":"cc721d0a-faab-45c5-8476-9aec206a65c3","Type":"ContainerStarted","Data":"075d0817d6a85f7907fa1c473e096950980e9eb54bb2ea3e5792e7a49a99191b"} Nov 03 20:30:03.255680 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:03.255654 4918 generic.go:356] "Generic (PLEG): container finished" podID="81a01d22-8738-4955-9dc5-73d769266ddc" containerID="b2251f29f239024fa30339275c141ba453d8cbcc30c7a56ce4959f9b0aa2c863" exitCode=60 Nov 03 20:30:03.255799 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:03.255730 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" event={"ID":"81a01d22-8738-4955-9dc5-73d769266ddc","Type":"ContainerDied","Data":"b2251f29f239024fa30339275c141ba453d8cbcc30c7a56ce4959f9b0aa2c863"} Nov 03 20:30:03.256214 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:03.256157 4918 scope.go:122] "RemoveContainer" containerID="b2251f29f239024fa30339275c141ba453d8cbcc30c7a56ce4959f9b0aa2c863" Nov 03 20:30:03.308117 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:03.308060 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw" podStartSLOduration=3.308041078 podStartE2EDuration="3.308041078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:03.300168065 +0000 UTC m=+1986.232035589" watchObservedRunningTime="2026-11-03 20:30:03.308041078 +0000 UTC m=+1986.239908592" Nov 03 20:30:03.318814 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:03.318757 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z" podStartSLOduration=3.31873918 podStartE2EDuration="3.31873918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:03.317339937 +0000 UTC m=+1986.249207451" watchObservedRunningTime="2026-11-03 20:30:03.31873918 +0000 UTC m=+1986.250606690" Nov 03 20:30:04.272380 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:04.272283 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" event={"ID":"81a01d22-8738-4955-9dc5-73d769266ddc","Type":"ContainerStarted","Data":"57f8aa66dc2c4801ba48176d4910f1497d74bcf740614a51e784c9da739f117e"} Nov 03 20:30:04.274447 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:04.274423 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" event={"ID":"c4cccb65-6424-4e0b-85bf-6f1fba8d5572","Type":"ContainerStarted","Data":"539dba20e3829570111e3f1d8413758b0949e74c42d0c968cbb680949cac9610"} Nov 03 20:30:04.276603 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:04.276576 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" event={"ID":"2228ed1d-945a-4b33-ba98-ee8b52cbf902","Type":"ContainerStarted","Data":"3769e40eedfdeb94aadcb582327d4b470919774bc062115286fe2d8b9e91c7d0"} Nov 03 20:30:04.280734 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:04.280701 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" event={"ID":"7a2653fa-4e55-48c8-a756-1f24c4c8412a","Type":"ContainerStarted","Data":"241d1ee6e1b8aa3ad7a522a43351e7a03ef702c8f382a99c9fe9d34ef8a9bf71"} Nov 03 20:30:04.283707 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:04.283679 4918 generic.go:356] "Generic (PLEG): container finished" podID="0832a6d3-76cd-4eb6-9879-c880f98a786f" containerID="c1faf9f51c7614053293b2572594078b34d9845179d416611441aa09f7aaff77" exitCode=60 Nov 03 20:30:04.283830 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:04.283789 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw" event={"ID":"0832a6d3-76cd-4eb6-9879-c880f98a786f","Type":"ContainerDied","Data":"c1faf9f51c7614053293b2572594078b34d9845179d416611441aa09f7aaff77"} Nov 03 20:30:04.283950 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:04.283868 4918 scope.go:122] "RemoveContainer" containerID="b75bab864f6bf2be38b6fc48517c3762ec2ca516b9a34a588f5c27a45e799e63" Nov 03 20:30:04.284229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:04.284165 4918 scope.go:122] "RemoveContainer" containerID="c1faf9f51c7614053293b2572594078b34d9845179d416611441aa09f7aaff77" Nov 03 20:30:04.284798 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:04.284766 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw_odh-ai-gateway-infra(0832a6d3-76cd-4eb6-9879-c880f98a786f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw" podUID="0832a6d3-76cd-4eb6-9879-c880f98a786f" Nov 03 20:30:04.292293 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:04.292261 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" event={"ID":"752114fb-ce1f-4ddb-8a69-75c8a44a364f","Type":"ContainerStarted","Data":"6e7b79c52a702ebe808fb0cb4d96832cb32b5e73e7564b56fff260c699c817d0"} Nov 03 20:30:04.295896 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:04.295866 4918 generic.go:356] "Generic (PLEG): container finished" podID="cc721d0a-faab-45c5-8476-9aec206a65c3" containerID="075d0817d6a85f7907fa1c473e096950980e9eb54bb2ea3e5792e7a49a99191b" exitCode=60 Nov 03 20:30:04.296021 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:04.295939 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z" event={"ID":"cc721d0a-faab-45c5-8476-9aec206a65c3","Type":"ContainerDied","Data":"075d0817d6a85f7907fa1c473e096950980e9eb54bb2ea3e5792e7a49a99191b"} Nov 03 20:30:04.296215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:04.296167 4918 scope.go:122] "RemoveContainer" containerID="075d0817d6a85f7907fa1c473e096950980e9eb54bb2ea3e5792e7a49a99191b" Nov 03 20:30:04.296509 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:04.296488 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z_odh-ai-gateway-infra(cc721d0a-faab-45c5-8476-9aec206a65c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z" podUID="cc721d0a-faab-45c5-8476-9aec206a65c3" Nov 03 20:30:04.306170 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:04.306143 4918 scope.go:122] "RemoveContainer" containerID="65ccf1e3f9723476cb5f6bdfc4bfa17e7db1479be50edb0f0984535733b9c9a3" Nov 03 20:30:05.307274 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.307241 4918 generic.go:356] "Generic (PLEG): container finished" podID="81a01d22-8738-4955-9dc5-73d769266ddc" containerID="57f8aa66dc2c4801ba48176d4910f1497d74bcf740614a51e784c9da739f117e" exitCode=60 Nov 03 20:30:05.307750 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.307326 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" event={"ID":"81a01d22-8738-4955-9dc5-73d769266ddc","Type":"ContainerDied","Data":"57f8aa66dc2c4801ba48176d4910f1497d74bcf740614a51e784c9da739f117e"} Nov 03 20:30:05.307750 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.307387 4918 scope.go:122] "RemoveContainer" containerID="b2251f29f239024fa30339275c141ba453d8cbcc30c7a56ce4959f9b0aa2c863" Nov 03 20:30:05.307750 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.307658 4918 scope.go:122] "RemoveContainer" containerID="57f8aa66dc2c4801ba48176d4910f1497d74bcf740614a51e784c9da739f117e" Nov 03 20:30:05.308224 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:05.308145 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx_odh-ai-gateway-infra(81a01d22-8738-4955-9dc5-73d769266ddc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" podUID="81a01d22-8738-4955-9dc5-73d769266ddc" Nov 03 20:30:05.316448 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.316422 4918 generic.go:356] "Generic (PLEG): container finished" podID="c4cccb65-6424-4e0b-85bf-6f1fba8d5572" containerID="539dba20e3829570111e3f1d8413758b0949e74c42d0c968cbb680949cac9610" exitCode=60 Nov 03 20:30:05.316586 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.316512 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" event={"ID":"c4cccb65-6424-4e0b-85bf-6f1fba8d5572","Type":"ContainerDied","Data":"539dba20e3829570111e3f1d8413758b0949e74c42d0c968cbb680949cac9610"} Nov 03 20:30:05.316879 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.316861 4918 scope.go:122] "RemoveContainer" containerID="539dba20e3829570111e3f1d8413758b0949e74c42d0c968cbb680949cac9610" Nov 03 20:30:05.317344 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:05.317315 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8_odh-ai-gateway-infra(c4cccb65-6424-4e0b-85bf-6f1fba8d5572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" podUID="c4cccb65-6424-4e0b-85bf-6f1fba8d5572" Nov 03 20:30:05.318617 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.318592 4918 generic.go:356] "Generic (PLEG): container finished" podID="2228ed1d-945a-4b33-ba98-ee8b52cbf902" containerID="3769e40eedfdeb94aadcb582327d4b470919774bc062115286fe2d8b9e91c7d0" exitCode=60 Nov 03 20:30:05.318705 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.318675 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" event={"ID":"2228ed1d-945a-4b33-ba98-ee8b52cbf902","Type":"ContainerDied","Data":"3769e40eedfdeb94aadcb582327d4b470919774bc062115286fe2d8b9e91c7d0"} Nov 03 20:30:05.318974 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.318959 4918 scope.go:122] "RemoveContainer" containerID="3769e40eedfdeb94aadcb582327d4b470919774bc062115286fe2d8b9e91c7d0" Nov 03 20:30:05.319415 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:05.319392 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92_odh-ai-gateway-infra(2228ed1d-945a-4b33-ba98-ee8b52cbf902)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" podUID="2228ed1d-945a-4b33-ba98-ee8b52cbf902" Nov 03 20:30:05.320695 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.320675 4918 generic.go:356] "Generic (PLEG): container finished" podID="7a2653fa-4e55-48c8-a756-1f24c4c8412a" containerID="241d1ee6e1b8aa3ad7a522a43351e7a03ef702c8f382a99c9fe9d34ef8a9bf71" exitCode=60 Nov 03 20:30:05.320794 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.320756 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" event={"ID":"7a2653fa-4e55-48c8-a756-1f24c4c8412a","Type":"ContainerDied","Data":"241d1ee6e1b8aa3ad7a522a43351e7a03ef702c8f382a99c9fe9d34ef8a9bf71"} Nov 03 20:30:05.321078 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.321058 4918 scope.go:122] "RemoveContainer" containerID="241d1ee6e1b8aa3ad7a522a43351e7a03ef702c8f382a99c9fe9d34ef8a9bf71" Nov 03 20:30:05.321490 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:05.321470 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t_odh-ai-gateway-infra(7a2653fa-4e55-48c8-a756-1f24c4c8412a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" podUID="7a2653fa-4e55-48c8-a756-1f24c4c8412a" Nov 03 20:30:05.323093 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.323072 4918 scope.go:122] "RemoveContainer" containerID="c1faf9f51c7614053293b2572594078b34d9845179d416611441aa09f7aaff77" Nov 03 20:30:05.323430 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:05.323410 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw_odh-ai-gateway-infra(0832a6d3-76cd-4eb6-9879-c880f98a786f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw" podUID="0832a6d3-76cd-4eb6-9879-c880f98a786f" Nov 03 20:30:05.324790 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.324772 4918 generic.go:356] "Generic (PLEG): container finished" podID="752114fb-ce1f-4ddb-8a69-75c8a44a364f" containerID="6e7b79c52a702ebe808fb0cb4d96832cb32b5e73e7564b56fff260c699c817d0" exitCode=60 Nov 03 20:30:05.324882 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.324797 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" event={"ID":"752114fb-ce1f-4ddb-8a69-75c8a44a364f","Type":"ContainerDied","Data":"6e7b79c52a702ebe808fb0cb4d96832cb32b5e73e7564b56fff260c699c817d0"} Nov 03 20:30:05.325126 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.325105 4918 scope.go:122] "RemoveContainer" containerID="6e7b79c52a702ebe808fb0cb4d96832cb32b5e73e7564b56fff260c699c817d0" Nov 03 20:30:05.325227 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.325208 4918 scope.go:122] "RemoveContainer" containerID="9b45eee7ebf9033090c2c04672113df13122dad71bd39393cfa5de3f187213cd" Nov 03 20:30:05.325493 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:05.325442 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2_odh-ai-gateway-infra(752114fb-ce1f-4ddb-8a69-75c8a44a364f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" podUID="752114fb-ce1f-4ddb-8a69-75c8a44a364f" Nov 03 20:30:05.326760 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.326739 4918 scope.go:122] "RemoveContainer" containerID="075d0817d6a85f7907fa1c473e096950980e9eb54bb2ea3e5792e7a49a99191b" Nov 03 20:30:05.327020 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:05.327001 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z_odh-ai-gateway-infra(cc721d0a-faab-45c5-8476-9aec206a65c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z" podUID="cc721d0a-faab-45c5-8476-9aec206a65c3" Nov 03 20:30:05.340299 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.340275 4918 scope.go:122] "RemoveContainer" containerID="010427b1b3dd1f09b8a070f250946f704ba293504d6cc776f7647d0a49f227ed" Nov 03 20:30:05.355390 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.355316 4918 scope.go:122] "RemoveContainer" containerID="2e88f28840110982deff2203ca9c230cf731dae4eac83c02caa76002bc3d9e00" Nov 03 20:30:05.370484 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:05.370464 4918 scope.go:122] "RemoveContainer" containerID="26ed325146a7ebe756accba282666ab8d05159ba46b0ada03767d56c270fd59e" Nov 03 20:30:06.337993 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:06.337961 4918 scope.go:122] "RemoveContainer" containerID="57f8aa66dc2c4801ba48176d4910f1497d74bcf740614a51e784c9da739f117e" Nov 03 20:30:06.338589 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:06.338561 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx_odh-ai-gateway-infra(81a01d22-8738-4955-9dc5-73d769266ddc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" podUID="81a01d22-8738-4955-9dc5-73d769266ddc" Nov 03 20:30:06.339712 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:06.339687 4918 scope.go:122] "RemoveContainer" containerID="539dba20e3829570111e3f1d8413758b0949e74c42d0c968cbb680949cac9610" Nov 03 20:30:06.340085 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:06.340059 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8_odh-ai-gateway-infra(c4cccb65-6424-4e0b-85bf-6f1fba8d5572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" podUID="c4cccb65-6424-4e0b-85bf-6f1fba8d5572" Nov 03 20:30:06.341451 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:06.341425 4918 scope.go:122] "RemoveContainer" containerID="3769e40eedfdeb94aadcb582327d4b470919774bc062115286fe2d8b9e91c7d0" Nov 03 20:30:06.341847 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:06.341818 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92_odh-ai-gateway-infra(2228ed1d-945a-4b33-ba98-ee8b52cbf902)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" podUID="2228ed1d-945a-4b33-ba98-ee8b52cbf902" Nov 03 20:30:06.343089 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:06.343066 4918 scope.go:122] "RemoveContainer" containerID="241d1ee6e1b8aa3ad7a522a43351e7a03ef702c8f382a99c9fe9d34ef8a9bf71" Nov 03 20:30:06.343437 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:06.343414 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t_odh-ai-gateway-infra(7a2653fa-4e55-48c8-a756-1f24c4c8412a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" podUID="7a2653fa-4e55-48c8-a756-1f24c4c8412a" Nov 03 20:30:06.344764 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:06.344748 4918 scope.go:122] "RemoveContainer" containerID="6e7b79c52a702ebe808fb0cb4d96832cb32b5e73e7564b56fff260c699c817d0" Nov 03 20:30:06.345028 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:06.345009 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2_odh-ai-gateway-infra(752114fb-ce1f-4ddb-8a69-75c8a44a364f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" podUID="752114fb-ce1f-4ddb-8a69-75c8a44a364f" Nov 03 20:30:08.592812 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:08.592777 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs"] Nov 03 20:30:08.599801 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:08.599772 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs" Nov 03 20:30:08.607227 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:08.605277 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:30:08.608007 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:08.607976 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs"] Nov 03 20:30:08.621158 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:08.621130 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fglhk\" (UniqueName: \"kubernetes.io/projected/67a9be3d-a3d7-450b-926f-96fa79503381-kube-api-access-fglhk\") pod \"maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs\" (UID: \"67a9be3d-a3d7-450b-926f-96fa79503381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs" Nov 03 20:30:08.621323 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:08.621294 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a9be3d-a3d7-450b-926f-96fa79503381-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs\" (UID: \"67a9be3d-a3d7-450b-926f-96fa79503381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs" Nov 03 20:30:08.621429 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:08.621409 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a9be3d-a3d7-450b-926f-96fa79503381-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs\" (UID: \"67a9be3d-a3d7-450b-926f-96fa79503381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs" Nov 03 20:30:08.723278 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:08.723230 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fglhk\" (UniqueName: \"kubernetes.io/projected/67a9be3d-a3d7-450b-926f-96fa79503381-kube-api-access-fglhk\") pod \"maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs\" (UID: \"67a9be3d-a3d7-450b-926f-96fa79503381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs" Nov 03 20:30:08.723495 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:08.723290 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a9be3d-a3d7-450b-926f-96fa79503381-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs\" (UID: \"67a9be3d-a3d7-450b-926f-96fa79503381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs" Nov 03 20:30:08.723495 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:08.723386 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a9be3d-a3d7-450b-926f-96fa79503381-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs\" (UID: \"67a9be3d-a3d7-450b-926f-96fa79503381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs" Nov 03 20:30:08.724088 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:08.724061 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a9be3d-a3d7-450b-926f-96fa79503381-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs\" (UID: \"67a9be3d-a3d7-450b-926f-96fa79503381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs" Nov 03 20:30:08.725894 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:08.725863 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a9be3d-a3d7-450b-926f-96fa79503381-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs\" (UID: \"67a9be3d-a3d7-450b-926f-96fa79503381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs" Nov 03 20:30:08.735687 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:08.735662 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fglhk\" (UniqueName: \"kubernetes.io/projected/67a9be3d-a3d7-450b-926f-96fa79503381-kube-api-access-fglhk\") pod \"maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs\" (UID: \"67a9be3d-a3d7-450b-926f-96fa79503381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs" Nov 03 20:30:08.923853 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:08.923631 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs" Nov 03 20:30:09.256268 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:09.256162 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs"] Nov 03 20:30:09.265440 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:30:09.265393 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a9be3d_a3d7_450b_926f_96fa79503381.slice/crio-117bbacd194659a1330686263d199dbedf7b0f39437e629b18ad1df9f6a6a0b1 WatchSource:0}: Error finding container 117bbacd194659a1330686263d199dbedf7b0f39437e629b18ad1df9f6a6a0b1: Status 404 returned error can't find the container with id 117bbacd194659a1330686263d199dbedf7b0f39437e629b18ad1df9f6a6a0b1 Nov 03 20:30:09.373832 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:09.373794 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs" event={"ID":"67a9be3d-a3d7-450b-926f-96fa79503381","Type":"ContainerStarted","Data":"117bbacd194659a1330686263d199dbedf7b0f39437e629b18ad1df9f6a6a0b1"} Nov 03 20:30:10.385138 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:10.385102 4918 generic.go:356] "Generic (PLEG): container finished" podID="67a9be3d-a3d7-450b-926f-96fa79503381" containerID="09e6b0aabba3904104d1daa6d4f6a4abf4997d7255b3b3f190b9860a58c0a5c2" exitCode=0 Nov 03 20:30:10.385707 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:10.385196 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs" event={"ID":"67a9be3d-a3d7-450b-926f-96fa79503381","Type":"ContainerDied","Data":"09e6b0aabba3904104d1daa6d4f6a4abf4997d7255b3b3f190b9860a58c0a5c2"} Nov 03 20:30:11.640859 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:11.640828 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs" Nov 03 20:30:11.695081 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:11.694993 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67a9be3d-a3d7-450b-926f-96fa79503381-kube-api-access-fglhk\" (UniqueName: \"kubernetes.io/projected/67a9be3d-a3d7-450b-926f-96fa79503381-kube-api-access-fglhk\") pod \"67a9be3d-a3d7-450b-926f-96fa79503381\" (UID: \"67a9be3d-a3d7-450b-926f-96fa79503381\") " Nov 03 20:30:11.695081 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:11.695051 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67a9be3d-a3d7-450b-926f-96fa79503381-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a9be3d-a3d7-450b-926f-96fa79503381-maas-api-xks-ca\") pod \"67a9be3d-a3d7-450b-926f-96fa79503381\" (UID: \"67a9be3d-a3d7-450b-926f-96fa79503381\") " Nov 03 20:30:11.695396 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:11.695220 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67a9be3d-a3d7-450b-926f-96fa79503381-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a9be3d-a3d7-450b-926f-96fa79503381-maas-api-openshift-ca\") pod \"67a9be3d-a3d7-450b-926f-96fa79503381\" (UID: \"67a9be3d-a3d7-450b-926f-96fa79503381\") " Nov 03 20:30:11.695694 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:11.695658 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a9be3d-a3d7-450b-926f-96fa79503381-maas-api-openshift-ca" pod "67a9be3d-a3d7-450b-926f-96fa79503381" (UID: "67a9be3d-a3d7-450b-926f-96fa79503381"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:11.696300 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:11.696272 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a9be3d-a3d7-450b-926f-96fa79503381-maas-api-openshift-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:11.697562 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:11.697532 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a9be3d-a3d7-450b-926f-96fa79503381-kube-api-access-fglhk" pod "67a9be3d-a3d7-450b-926f-96fa79503381" (UID: "67a9be3d-a3d7-450b-926f-96fa79503381"). InnerVolumeSpecName "kube-api-access-fglhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:11.698020 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:11.697986 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a9be3d-a3d7-450b-926f-96fa79503381-maas-api-xks-ca" pod "67a9be3d-a3d7-450b-926f-96fa79503381" (UID: "67a9be3d-a3d7-450b-926f-96fa79503381"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:11.798292 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:11.798262 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fglhk\" (UniqueName: \"kubernetes.io/projected/67a9be3d-a3d7-450b-926f-96fa79503381-kube-api-access-fglhk\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:11.798292 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:11.798290 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a9be3d-a3d7-450b-926f-96fa79503381-maas-api-xks-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:12.403336 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:12.403295 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs" event={"ID":"67a9be3d-a3d7-450b-926f-96fa79503381","Type":"ContainerDied","Data":"117bbacd194659a1330686263d199dbedf7b0f39437e629b18ad1df9f6a6a0b1"} Nov 03 20:30:12.403510 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:12.403355 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="117bbacd194659a1330686263d199dbedf7b0f39437e629b18ad1df9f6a6a0b1" Nov 03 20:30:12.403510 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:12.403304 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs" Nov 03 20:30:13.342138 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:13.342104 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v"] Nov 03 20:30:13.351030 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:13.350999 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt"] Nov 03 20:30:15.207808 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:15.207774 4918 scope.go:122] "RemoveContainer" containerID="075d0817d6a85f7907fa1c473e096950980e9eb54bb2ea3e5792e7a49a99191b" Nov 03 20:30:16.207974 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:16.207936 4918 scope.go:122] "RemoveContainer" containerID="3769e40eedfdeb94aadcb582327d4b470919774bc062115286fe2d8b9e91c7d0" Nov 03 20:30:16.208459 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:16.208279 4918 scope.go:122] "RemoveContainer" containerID="c1faf9f51c7614053293b2572594078b34d9845179d416611441aa09f7aaff77" Nov 03 20:30:16.208459 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:16.208375 4918 scope.go:122] "RemoveContainer" containerID="539dba20e3829570111e3f1d8413758b0949e74c42d0c968cbb680949cac9610" Nov 03 20:30:16.208547 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:16.208469 4918 scope.go:122] "RemoveContainer" containerID="241d1ee6e1b8aa3ad7a522a43351e7a03ef702c8f382a99c9fe9d34ef8a9bf71" Nov 03 20:30:16.439638 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:16.439608 4918 generic.go:356] "Generic (PLEG): container finished" podID="cc721d0a-faab-45c5-8476-9aec206a65c3" containerID="c1448996b1217df2dd621f29f65317b693057c4a6e9224d1eff541c02ddaaa3c" exitCode=60 Nov 03 20:30:16.439787 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:16.439686 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z" event={"ID":"cc721d0a-faab-45c5-8476-9aec206a65c3","Type":"ContainerDied","Data":"c1448996b1217df2dd621f29f65317b693057c4a6e9224d1eff541c02ddaaa3c"} Nov 03 20:30:16.439787 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:16.439782 4918 scope.go:122] "RemoveContainer" containerID="075d0817d6a85f7907fa1c473e096950980e9eb54bb2ea3e5792e7a49a99191b" Nov 03 20:30:16.440351 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:16.440326 4918 scope.go:122] "RemoveContainer" containerID="c1448996b1217df2dd621f29f65317b693057c4a6e9224d1eff541c02ddaaa3c" Nov 03 20:30:16.440804 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:16.440779 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z_odh-ai-gateway-infra(cc721d0a-faab-45c5-8476-9aec206a65c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z" podUID="cc721d0a-faab-45c5-8476-9aec206a65c3" Nov 03 20:30:17.212725 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:17.212645 4918 scope.go:122] "RemoveContainer" containerID="57f8aa66dc2c4801ba48176d4910f1497d74bcf740614a51e784c9da739f117e" Nov 03 20:30:17.480578 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:17.480501 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z"] Nov 03 20:30:17.489634 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:17.489601 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" event={"ID":"7a2653fa-4e55-48c8-a756-1f24c4c8412a","Type":"ContainerStarted","Data":"3b0262f9d5185d414ad1c88a206fe4c85180763e31de3cde6b0fb03d31afde6a"} Nov 03 20:30:17.490262 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:17.490235 4918 scope.go:122] "RemoveContainer" containerID="3b0262f9d5185d414ad1c88a206fe4c85180763e31de3cde6b0fb03d31afde6a" Nov 03 20:30:17.492826 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:17.492789 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t_odh-ai-gateway-infra(7a2653fa-4e55-48c8-a756-1f24c4c8412a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" podUID="7a2653fa-4e55-48c8-a756-1f24c4c8412a" Nov 03 20:30:17.492945 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:17.492853 4918 generic.go:356] "Generic (PLEG): container finished" podID="0832a6d3-76cd-4eb6-9879-c880f98a786f" containerID="e0b485b35002970bdced902a5c799128a7f947b106fdd6519ccb09cea5b58275" exitCode=60 Nov 03 20:30:17.492945 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:17.492922 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw" event={"ID":"0832a6d3-76cd-4eb6-9879-c880f98a786f","Type":"ContainerDied","Data":"e0b485b35002970bdced902a5c799128a7f947b106fdd6519ccb09cea5b58275"} Nov 03 20:30:17.493047 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:17.492968 4918 scope.go:122] "RemoveContainer" containerID="c1faf9f51c7614053293b2572594078b34d9845179d416611441aa09f7aaff77" Nov 03 20:30:17.493546 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:17.493516 4918 scope.go:122] "RemoveContainer" containerID="e0b485b35002970bdced902a5c799128a7f947b106fdd6519ccb09cea5b58275" Nov 03 20:30:17.494287 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:17.493960 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw_odh-ai-gateway-infra(0832a6d3-76cd-4eb6-9879-c880f98a786f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw" podUID="0832a6d3-76cd-4eb6-9879-c880f98a786f" Nov 03 20:30:17.497703 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:17.497673 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" event={"ID":"c4cccb65-6424-4e0b-85bf-6f1fba8d5572","Type":"ContainerStarted","Data":"3224c08ecef22fb81b88b822137206aa061c87d00daaf34da14c76670cb534f5"} Nov 03 20:30:17.498168 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:17.498143 4918 scope.go:122] "RemoveContainer" containerID="3224c08ecef22fb81b88b822137206aa061c87d00daaf34da14c76670cb534f5" Nov 03 20:30:17.498602 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:17.498566 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8_odh-ai-gateway-infra(c4cccb65-6424-4e0b-85bf-6f1fba8d5572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" podUID="c4cccb65-6424-4e0b-85bf-6f1fba8d5572" Nov 03 20:30:17.500229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:17.500169 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" event={"ID":"2228ed1d-945a-4b33-ba98-ee8b52cbf902","Type":"ContainerStarted","Data":"9c9ac313aaa47c6896d49923017cc226e47c85c17db2769d67a32cb8a4f2702b"} Nov 03 20:30:17.735085 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:17.735060 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z" Nov 03 20:30:17.839973 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:17.839939 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc721d0a-faab-45c5-8476-9aec206a65c3-kube-api-access-cffzr\" (UniqueName: \"kubernetes.io/projected/cc721d0a-faab-45c5-8476-9aec206a65c3-kube-api-access-cffzr\") pod \"cc721d0a-faab-45c5-8476-9aec206a65c3\" (UID: \"cc721d0a-faab-45c5-8476-9aec206a65c3\") " Nov 03 20:30:17.840165 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:17.840148 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc721d0a-faab-45c5-8476-9aec206a65c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc721d0a-faab-45c5-8476-9aec206a65c3-maas-api-service-ca\") pod \"cc721d0a-faab-45c5-8476-9aec206a65c3\" (UID: \"cc721d0a-faab-45c5-8476-9aec206a65c3\") " Nov 03 20:30:17.840547 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:17.840513 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc721d0a-faab-45c5-8476-9aec206a65c3-maas-api-service-ca" pod "cc721d0a-faab-45c5-8476-9aec206a65c3" (UID: "cc721d0a-faab-45c5-8476-9aec206a65c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:17.841124 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:17.841105 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc721d0a-faab-45c5-8476-9aec206a65c3-maas-api-service-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:17.842148 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:17.842124 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc721d0a-faab-45c5-8476-9aec206a65c3-kube-api-access-cffzr" pod "cc721d0a-faab-45c5-8476-9aec206a65c3" (UID: "cc721d0a-faab-45c5-8476-9aec206a65c3"). InnerVolumeSpecName "kube-api-access-cffzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:17.943042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:17.943014 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cffzr\" (UniqueName: \"kubernetes.io/projected/cc721d0a-faab-45c5-8476-9aec206a65c3-kube-api-access-cffzr\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:18.207996 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.207962 4918 scope.go:122] "RemoveContainer" containerID="6e7b79c52a702ebe808fb0cb4d96832cb32b5e73e7564b56fff260c699c817d0" Nov 03 20:30:18.511102 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.511014 4918 generic.go:356] "Generic (PLEG): container finished" podID="c4cccb65-6424-4e0b-85bf-6f1fba8d5572" containerID="3224c08ecef22fb81b88b822137206aa061c87d00daaf34da14c76670cb534f5" exitCode=60 Nov 03 20:30:18.511102 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.511083 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" event={"ID":"c4cccb65-6424-4e0b-85bf-6f1fba8d5572","Type":"ContainerDied","Data":"3224c08ecef22fb81b88b822137206aa061c87d00daaf34da14c76670cb534f5"} Nov 03 20:30:18.511625 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.511155 4918 scope.go:122] "RemoveContainer" containerID="539dba20e3829570111e3f1d8413758b0949e74c42d0c968cbb680949cac9610" Nov 03 20:30:18.511684 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.511635 4918 scope.go:122] "RemoveContainer" containerID="3224c08ecef22fb81b88b822137206aa061c87d00daaf34da14c76670cb534f5" Nov 03 20:30:18.512095 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:18.512062 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8_odh-ai-gateway-infra(c4cccb65-6424-4e0b-85bf-6f1fba8d5572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" podUID="c4cccb65-6424-4e0b-85bf-6f1fba8d5572" Nov 03 20:30:18.513497 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.513473 4918 generic.go:356] "Generic (PLEG): container finished" podID="2228ed1d-945a-4b33-ba98-ee8b52cbf902" containerID="9c9ac313aaa47c6896d49923017cc226e47c85c17db2769d67a32cb8a4f2702b" exitCode=60 Nov 03 20:30:18.513599 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.513540 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" event={"ID":"2228ed1d-945a-4b33-ba98-ee8b52cbf902","Type":"ContainerDied","Data":"9c9ac313aaa47c6896d49923017cc226e47c85c17db2769d67a32cb8a4f2702b"} Nov 03 20:30:18.513963 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.513945 4918 scope.go:122] "RemoveContainer" containerID="9c9ac313aaa47c6896d49923017cc226e47c85c17db2769d67a32cb8a4f2702b" Nov 03 20:30:18.514361 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:18.514337 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92_odh-ai-gateway-infra(2228ed1d-945a-4b33-ba98-ee8b52cbf902)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" podUID="2228ed1d-945a-4b33-ba98-ee8b52cbf902" Nov 03 20:30:18.516598 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.516577 4918 generic.go:356] "Generic (PLEG): container finished" podID="7a2653fa-4e55-48c8-a756-1f24c4c8412a" containerID="3b0262f9d5185d414ad1c88a206fe4c85180763e31de3cde6b0fb03d31afde6a" exitCode=60 Nov 03 20:30:18.516684 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.516645 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" event={"ID":"7a2653fa-4e55-48c8-a756-1f24c4c8412a","Type":"ContainerDied","Data":"3b0262f9d5185d414ad1c88a206fe4c85180763e31de3cde6b0fb03d31afde6a"} Nov 03 20:30:18.516979 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.516961 4918 scope.go:122] "RemoveContainer" containerID="3b0262f9d5185d414ad1c88a206fe4c85180763e31de3cde6b0fb03d31afde6a" Nov 03 20:30:18.517291 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:18.517269 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t_odh-ai-gateway-infra(7a2653fa-4e55-48c8-a756-1f24c4c8412a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" podUID="7a2653fa-4e55-48c8-a756-1f24c4c8412a" Nov 03 20:30:18.523121 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.523098 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z" Nov 03 20:30:18.523266 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.523147 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z" event={"ID":"cc721d0a-faab-45c5-8476-9aec206a65c3","Type":"ContainerDied","Data":"d8e2f0b0ce9b8a01050e6eb3337bff13b53641884a3f6f6e9cc37b0ebb475fc7"} Nov 03 20:30:18.525759 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.525735 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" event={"ID":"81a01d22-8738-4955-9dc5-73d769266ddc","Type":"ContainerStarted","Data":"2976660f5d170eb1f6cd42f5abf03479b95554c231e4924ba56b57795f5b4811"} Nov 03 20:30:18.534550 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.534525 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t"] Nov 03 20:30:18.546803 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.546777 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw"] Nov 03 20:30:18.582223 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.581384 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92"] Nov 03 20:30:18.593728 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.593703 4918 scope.go:122] "RemoveContainer" containerID="3769e40eedfdeb94aadcb582327d4b470919774bc062115286fe2d8b9e91c7d0" Nov 03 20:30:18.593963 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.593922 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8"] Nov 03 20:30:18.612337 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.612314 4918 scope.go:122] "RemoveContainer" containerID="241d1ee6e1b8aa3ad7a522a43351e7a03ef702c8f382a99c9fe9d34ef8a9bf71" Nov 03 20:30:18.640615 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.640579 4918 scope.go:122] "RemoveContainer" containerID="c1448996b1217df2dd621f29f65317b693057c4a6e9224d1eff541c02ddaaa3c" Nov 03 20:30:18.678520 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.677847 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z"] Nov 03 20:30:18.684833 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.684801 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b35c30-29895630-qbr6z"] Nov 03 20:30:18.772918 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.772841 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw" Nov 03 20:30:18.890517 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.890485 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0832a6d3-76cd-4eb6-9879-c880f98a786f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0832a6d3-76cd-4eb6-9879-c880f98a786f-maas-api-service-ca\") pod \"0832a6d3-76cd-4eb6-9879-c880f98a786f\" (UID: \"0832a6d3-76cd-4eb6-9879-c880f98a786f\") " Nov 03 20:30:18.890712 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.890587 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0832a6d3-76cd-4eb6-9879-c880f98a786f-kube-api-access-6wbp5\" (UniqueName: \"kubernetes.io/projected/0832a6d3-76cd-4eb6-9879-c880f98a786f-kube-api-access-6wbp5\") pod \"0832a6d3-76cd-4eb6-9879-c880f98a786f\" (UID: \"0832a6d3-76cd-4eb6-9879-c880f98a786f\") " Nov 03 20:30:18.890905 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.890878 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0832a6d3-76cd-4eb6-9879-c880f98a786f-maas-api-service-ca" pod "0832a6d3-76cd-4eb6-9879-c880f98a786f" (UID: "0832a6d3-76cd-4eb6-9879-c880f98a786f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:18.891326 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.891304 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0832a6d3-76cd-4eb6-9879-c880f98a786f-maas-api-service-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:18.892794 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.892763 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0832a6d3-76cd-4eb6-9879-c880f98a786f-kube-api-access-6wbp5" pod "0832a6d3-76cd-4eb6-9879-c880f98a786f" (UID: "0832a6d3-76cd-4eb6-9879-c880f98a786f"). InnerVolumeSpecName "kube-api-access-6wbp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:18.993988 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:18.993955 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wbp5\" (UniqueName: \"kubernetes.io/projected/0832a6d3-76cd-4eb6-9879-c880f98a786f-kube-api-access-6wbp5\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:19.240140 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:19.237206 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc721d0a-faab-45c5-8476-9aec206a65c3" path="/var/lib/kubelet/pods/cc721d0a-faab-45c5-8476-9aec206a65c3/volumes" Nov 03 20:30:19.535099 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:19.535014 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw" Nov 03 20:30:19.535099 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:19.535041 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw" event={"ID":"0832a6d3-76cd-4eb6-9879-c880f98a786f","Type":"ContainerDied","Data":"fb2ce4c6baebc0075dfa4eb64fb0d88aa88e0b6497cd6a3f2f8faf494eb29265"} Nov 03 20:30:19.535636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:19.535107 4918 scope.go:122] "RemoveContainer" containerID="e0b485b35002970bdced902a5c799128a7f947b106fdd6519ccb09cea5b58275" Nov 03 20:30:19.537488 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:19.537462 4918 generic.go:356] "Generic (PLEG): container finished" podID="752114fb-ce1f-4ddb-8a69-75c8a44a364f" containerID="6d7d51cb96db028542031bb50e168f4bda7974ca906754abb1285c95ba836c92" exitCode=60 Nov 03 20:30:19.537645 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:19.537515 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" event={"ID":"752114fb-ce1f-4ddb-8a69-75c8a44a364f","Type":"ContainerDied","Data":"6d7d51cb96db028542031bb50e168f4bda7974ca906754abb1285c95ba836c92"} Nov 03 20:30:19.538599 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:19.538124 4918 scope.go:122] "RemoveContainer" containerID="6d7d51cb96db028542031bb50e168f4bda7974ca906754abb1285c95ba836c92" Nov 03 20:30:19.538599 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:19.538561 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2_odh-ai-gateway-infra(752114fb-ce1f-4ddb-8a69-75c8a44a364f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" podUID="752114fb-ce1f-4ddb-8a69-75c8a44a364f" Nov 03 20:30:19.541148 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:19.541129 4918 generic.go:356] "Generic (PLEG): container finished" podID="81a01d22-8738-4955-9dc5-73d769266ddc" containerID="2976660f5d170eb1f6cd42f5abf03479b95554c231e4924ba56b57795f5b4811" exitCode=60 Nov 03 20:30:19.541253 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:19.541204 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" event={"ID":"81a01d22-8738-4955-9dc5-73d769266ddc","Type":"ContainerDied","Data":"2976660f5d170eb1f6cd42f5abf03479b95554c231e4924ba56b57795f5b4811"} Nov 03 20:30:19.541625 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:19.541603 4918 scope.go:122] "RemoveContainer" containerID="2976660f5d170eb1f6cd42f5abf03479b95554c231e4924ba56b57795f5b4811" Nov 03 20:30:19.541967 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:19.541940 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx_odh-ai-gateway-infra(81a01d22-8738-4955-9dc5-73d769266ddc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" podUID="81a01d22-8738-4955-9dc5-73d769266ddc" Nov 03 20:30:19.547924 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:19.547906 4918 scope.go:122] "RemoveContainer" containerID="6e7b79c52a702ebe808fb0cb4d96832cb32b5e73e7564b56fff260c699c817d0" Nov 03 20:30:19.565163 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:19.565119 4918 scope.go:122] "RemoveContainer" containerID="57f8aa66dc2c4801ba48176d4910f1497d74bcf740614a51e784c9da739f117e" Nov 03 20:30:19.676829 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:19.676786 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx"] Nov 03 20:30:19.766279 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:19.766246 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw"] Nov 03 20:30:19.774360 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:19.774332 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c2c601-29895630-dphkw"] Nov 03 20:30:19.881537 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:19.881514 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" Nov 03 20:30:19.931417 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:19.931382 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" Nov 03 20:30:19.964818 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:19.964519 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" Nov 03 20:30:20.031704 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.031670 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2228ed1d-945a-4b33-ba98-ee8b52cbf902-kube-api-access-6kg8h\" (UniqueName: \"kubernetes.io/projected/2228ed1d-945a-4b33-ba98-ee8b52cbf902-kube-api-access-6kg8h\") pod \"2228ed1d-945a-4b33-ba98-ee8b52cbf902\" (UID: \"2228ed1d-945a-4b33-ba98-ee8b52cbf902\") " Nov 03 20:30:20.032401 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.032362 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2653fa-4e55-48c8-a756-1f24c4c8412a-kube-api-access-mxft2\" (UniqueName: \"kubernetes.io/projected/7a2653fa-4e55-48c8-a756-1f24c4c8412a-kube-api-access-mxft2\") pod \"7a2653fa-4e55-48c8-a756-1f24c4c8412a\" (UID: \"7a2653fa-4e55-48c8-a756-1f24c4c8412a\") " Nov 03 20:30:20.032882 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.032862 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a2653fa-4e55-48c8-a756-1f24c4c8412a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a2653fa-4e55-48c8-a756-1f24c4c8412a-maas-api-service-ca\") pod \"7a2653fa-4e55-48c8-a756-1f24c4c8412a\" (UID: \"7a2653fa-4e55-48c8-a756-1f24c4c8412a\") " Nov 03 20:30:20.033692 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.033128 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2653fa-4e55-48c8-a756-1f24c4c8412a-maas-api-service-ca" pod "7a2653fa-4e55-48c8-a756-1f24c4c8412a" (UID: "7a2653fa-4e55-48c8-a756-1f24c4c8412a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:20.033782 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.033671 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2228ed1d-945a-4b33-ba98-ee8b52cbf902-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2228ed1d-945a-4b33-ba98-ee8b52cbf902-maas-api-service-ca\") pod \"2228ed1d-945a-4b33-ba98-ee8b52cbf902\" (UID: \"2228ed1d-945a-4b33-ba98-ee8b52cbf902\") " Nov 03 20:30:20.034171 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.034147 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2228ed1d-945a-4b33-ba98-ee8b52cbf902-maas-api-service-ca" pod "2228ed1d-945a-4b33-ba98-ee8b52cbf902" (UID: "2228ed1d-945a-4b33-ba98-ee8b52cbf902"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:20.034774 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.034751 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a2653fa-4e55-48c8-a756-1f24c4c8412a-maas-api-service-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.034908 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.034892 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2228ed1d-945a-4b33-ba98-ee8b52cbf902-maas-api-service-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.036287 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.036263 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2228ed1d-945a-4b33-ba98-ee8b52cbf902-kube-api-access-6kg8h" pod "2228ed1d-945a-4b33-ba98-ee8b52cbf902" (UID: "2228ed1d-945a-4b33-ba98-ee8b52cbf902"). InnerVolumeSpecName "kube-api-access-6kg8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:20.036582 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.036566 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2653fa-4e55-48c8-a756-1f24c4c8412a-kube-api-access-mxft2" pod "7a2653fa-4e55-48c8-a756-1f24c4c8412a" (UID: "7a2653fa-4e55-48c8-a756-1f24c4c8412a"). InnerVolumeSpecName "kube-api-access-mxft2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:20.136016 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.135932 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4cccb65-6424-4e0b-85bf-6f1fba8d5572-kube-api-access-6st28\" (UniqueName: \"kubernetes.io/projected/c4cccb65-6424-4e0b-85bf-6f1fba8d5572-kube-api-access-6st28\") pod \"c4cccb65-6424-4e0b-85bf-6f1fba8d5572\" (UID: \"c4cccb65-6424-4e0b-85bf-6f1fba8d5572\") " Nov 03 20:30:20.136173 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.136039 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4cccb65-6424-4e0b-85bf-6f1fba8d5572-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4cccb65-6424-4e0b-85bf-6f1fba8d5572-maas-api-service-ca\") pod \"c4cccb65-6424-4e0b-85bf-6f1fba8d5572\" (UID: \"c4cccb65-6424-4e0b-85bf-6f1fba8d5572\") " Nov 03 20:30:20.136450 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.136426 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4cccb65-6424-4e0b-85bf-6f1fba8d5572-maas-api-service-ca" pod "c4cccb65-6424-4e0b-85bf-6f1fba8d5572" (UID: "c4cccb65-6424-4e0b-85bf-6f1fba8d5572"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:20.136976 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.136953 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4cccb65-6424-4e0b-85bf-6f1fba8d5572-maas-api-service-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.136976 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.136977 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kg8h\" (UniqueName: \"kubernetes.io/projected/2228ed1d-945a-4b33-ba98-ee8b52cbf902-kube-api-access-6kg8h\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.137117 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.136990 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxft2\" (UniqueName: \"kubernetes.io/projected/7a2653fa-4e55-48c8-a756-1f24c4c8412a-kube-api-access-mxft2\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.138534 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.138509 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4cccb65-6424-4e0b-85bf-6f1fba8d5572-kube-api-access-6st28" pod "c4cccb65-6424-4e0b-85bf-6f1fba8d5572" (UID: "c4cccb65-6424-4e0b-85bf-6f1fba8d5572"). InnerVolumeSpecName "kube-api-access-6st28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:20.238218 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.238163 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6st28\" (UniqueName: \"kubernetes.io/projected/c4cccb65-6424-4e0b-85bf-6f1fba8d5572-kube-api-access-6st28\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.559547 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.559516 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" Nov 03 20:30:20.559960 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.559509 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8" event={"ID":"c4cccb65-6424-4e0b-85bf-6f1fba8d5572","Type":"ContainerDied","Data":"31f7e4ac11f13b6aa6271a054764a772cd22402723971acdb386c34208f45b95"} Nov 03 20:30:20.559960 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.559643 4918 scope.go:122] "RemoveContainer" containerID="3224c08ecef22fb81b88b822137206aa061c87d00daaf34da14c76670cb534f5" Nov 03 20:30:20.561486 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.561455 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" Nov 03 20:30:20.561611 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.561496 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92" event={"ID":"2228ed1d-945a-4b33-ba98-ee8b52cbf902","Type":"ContainerDied","Data":"0f52b89a9b479f0f4da6a5b3eeda23c8407ddd76aae89e1712ab34687d5cb5bb"} Nov 03 20:30:20.567758 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.567738 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" Nov 03 20:30:20.567883 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.567773 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t" event={"ID":"7a2653fa-4e55-48c8-a756-1f24c4c8412a","Type":"ContainerDied","Data":"75236c2145aea45de3fa5ef9b1f9524847a2ef90b177ea5caf9ddaad4ce85071"} Nov 03 20:30:20.572415 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.572396 4918 scope.go:122] "RemoveContainer" containerID="9c9ac313aaa47c6896d49923017cc226e47c85c17db2769d67a32cb8a4f2702b" Nov 03 20:30:20.584679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.584653 4918 scope.go:122] "RemoveContainer" containerID="3b0262f9d5185d414ad1c88a206fe4c85180763e31de3cde6b0fb03d31afde6a" Nov 03 20:30:20.693936 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.693648 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92"] Nov 03 20:30:20.701455 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.700168 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895630-64v92"] Nov 03 20:30:20.734894 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.731384 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2"] Nov 03 20:30:20.745033 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.744992 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8"] Nov 03 20:30:20.752473 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.749899 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c60d2a-29895630-w7lg8"] Nov 03 20:30:20.780215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.777853 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t"] Nov 03 20:30:20.780215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.778438 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0adda-29895630-c2b6t"] Nov 03 20:30:20.867000 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.866969 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" Nov 03 20:30:20.951881 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.951850 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a01d22-8738-4955-9dc5-73d769266ddc-kube-api-access-64kct\" (UniqueName: \"kubernetes.io/projected/81a01d22-8738-4955-9dc5-73d769266ddc-kube-api-access-64kct\") pod \"81a01d22-8738-4955-9dc5-73d769266ddc\" (UID: \"81a01d22-8738-4955-9dc5-73d769266ddc\") " Nov 03 20:30:20.952123 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.952106 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a01d22-8738-4955-9dc5-73d769266ddc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81a01d22-8738-4955-9dc5-73d769266ddc-maas-api-service-ca\") pod \"81a01d22-8738-4955-9dc5-73d769266ddc\" (UID: \"81a01d22-8738-4955-9dc5-73d769266ddc\") " Nov 03 20:30:20.952619 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.952589 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a01d22-8738-4955-9dc5-73d769266ddc-maas-api-service-ca" pod "81a01d22-8738-4955-9dc5-73d769266ddc" (UID: "81a01d22-8738-4955-9dc5-73d769266ddc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:20.953002 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.952981 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81a01d22-8738-4955-9dc5-73d769266ddc-maas-api-service-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.954382 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.954352 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a01d22-8738-4955-9dc5-73d769266ddc-kube-api-access-64kct" pod "81a01d22-8738-4955-9dc5-73d769266ddc" (UID: "81a01d22-8738-4955-9dc5-73d769266ddc"). InnerVolumeSpecName "kube-api-access-64kct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:20.960811 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:20.960784 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" Nov 03 20:30:21.053854 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.053765 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752114fb-ce1f-4ddb-8a69-75c8a44a364f-kube-api-access-ghf9k\" (UniqueName: \"kubernetes.io/projected/752114fb-ce1f-4ddb-8a69-75c8a44a364f-kube-api-access-ghf9k\") pod \"752114fb-ce1f-4ddb-8a69-75c8a44a364f\" (UID: \"752114fb-ce1f-4ddb-8a69-75c8a44a364f\") " Nov 03 20:30:21.054016 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.053978 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/752114fb-ce1f-4ddb-8a69-75c8a44a364f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/752114fb-ce1f-4ddb-8a69-75c8a44a364f-maas-api-service-ca\") pod \"752114fb-ce1f-4ddb-8a69-75c8a44a364f\" (UID: \"752114fb-ce1f-4ddb-8a69-75c8a44a364f\") " Nov 03 20:30:21.054455 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.054406 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752114fb-ce1f-4ddb-8a69-75c8a44a364f-maas-api-service-ca" pod "752114fb-ce1f-4ddb-8a69-75c8a44a364f" (UID: "752114fb-ce1f-4ddb-8a69-75c8a44a364f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:21.054912 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.054880 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/752114fb-ce1f-4ddb-8a69-75c8a44a364f-maas-api-service-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.054912 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.054909 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64kct\" (UniqueName: \"kubernetes.io/projected/81a01d22-8738-4955-9dc5-73d769266ddc-kube-api-access-64kct\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.056647 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.056621 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752114fb-ce1f-4ddb-8a69-75c8a44a364f-kube-api-access-ghf9k" pod "752114fb-ce1f-4ddb-8a69-75c8a44a364f" (UID: "752114fb-ce1f-4ddb-8a69-75c8a44a364f"). InnerVolumeSpecName "kube-api-access-ghf9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:21.157055 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.157012 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghf9k\" (UniqueName: \"kubernetes.io/projected/752114fb-ce1f-4ddb-8a69-75c8a44a364f-kube-api-access-ghf9k\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.219023 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.218991 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0832a6d3-76cd-4eb6-9879-c880f98a786f" path="/var/lib/kubelet/pods/0832a6d3-76cd-4eb6-9879-c880f98a786f/volumes" Nov 03 20:30:21.219527 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.219507 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2228ed1d-945a-4b33-ba98-ee8b52cbf902" path="/var/lib/kubelet/pods/2228ed1d-945a-4b33-ba98-ee8b52cbf902/volumes" Nov 03 20:30:21.219962 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.219946 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2653fa-4e55-48c8-a756-1f24c4c8412a" path="/var/lib/kubelet/pods/7a2653fa-4e55-48c8-a756-1f24c4c8412a/volumes" Nov 03 20:30:21.220682 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.220666 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4cccb65-6424-4e0b-85bf-6f1fba8d5572" path="/var/lib/kubelet/pods/c4cccb65-6424-4e0b-85bf-6f1fba8d5572/volumes" Nov 03 20:30:21.585487 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.585460 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" Nov 03 20:30:21.585916 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.585503 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2" event={"ID":"752114fb-ce1f-4ddb-8a69-75c8a44a364f","Type":"ContainerDied","Data":"b3b702d9b58cdf26b12264af31b713a626a2606dfd19179ba2569f8afbce5693"} Nov 03 20:30:21.585916 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.585578 4918 scope.go:122] "RemoveContainer" containerID="6d7d51cb96db028542031bb50e168f4bda7974ca906754abb1285c95ba836c92" Nov 03 20:30:21.587641 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.587612 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" Nov 03 20:30:21.587729 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.587672 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx" event={"ID":"81a01d22-8738-4955-9dc5-73d769266ddc","Type":"ContainerDied","Data":"6696f9fc96312b0f37156a8732ca0f051fc80158a1a61d53d16171121bdcc0db"} Nov 03 20:30:21.598602 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.598580 4918 scope.go:122] "RemoveContainer" containerID="2976660f5d170eb1f6cd42f5abf03479b95554c231e4924ba56b57795f5b4811" Nov 03 20:30:21.620664 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.620619 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2"] Nov 03 20:30:21.625382 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.625354 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ba8f5b-29895630-qccv2"] Nov 03 20:30:21.639479 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.639453 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx"] Nov 03 20:30:21.650981 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:21.650947 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895630-9mxgx"] Nov 03 20:30:23.220108 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:23.220072 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="752114fb-ce1f-4ddb-8a69-75c8a44a364f" path="/var/lib/kubelet/pods/752114fb-ce1f-4ddb-8a69-75c8a44a364f/volumes" Nov 03 20:30:23.220614 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:23.220599 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a01d22-8738-4955-9dc5-73d769266ddc" path="/var/lib/kubelet/pods/81a01d22-8738-4955-9dc5-73d769266ddc/volumes" Nov 03 20:30:28.210600 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:28.210559 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:30:28.364971 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.364933 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" containerID="cri-o://3e4f1c5f1cac945da0445311a715a360e20769378f7435fe6c6ba4cffa7fe07d" gracePeriod=45 Nov 03 20:30:28.373558 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.373519 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" containerID="cri-o://74fe3f36ca790f86d20f06f6e23b3e6a09da1850873af80fd6696dfb3b3bb417" gracePeriod=45 Nov 03 20:30:28.502925 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.502886 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": dial tcp 10.132.0.62:8000: connect: connection refused" Nov 03 20:30:28.735549 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.735513 4918 generic.go:356] "Generic (PLEG): container finished" podID="b14fd53e-9193-44a0-9693-446fe067a841" containerID="74fe3f36ca790f86d20f06f6e23b3e6a09da1850873af80fd6696dfb3b3bb417" exitCode=0 Nov 03 20:30:28.735714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.735641 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" event={"ID":"b14fd53e-9193-44a0-9693-446fe067a841","Type":"ContainerDied","Data":"74fe3f36ca790f86d20f06f6e23b3e6a09da1850873af80fd6696dfb3b3bb417"} Nov 03 20:30:28.747914 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.747877 4918 generic.go:356] "Generic (PLEG): container finished" podID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerID="3e4f1c5f1cac945da0445311a715a360e20769378f7435fe6c6ba4cffa7fe07d" exitCode=0 Nov 03 20:30:28.748043 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.747950 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" event={"ID":"6f5ce248-0c9e-4f39-9130-1b2aad5beb65","Type":"ContainerDied","Data":"3e4f1c5f1cac945da0445311a715a360e20769378f7435fe6c6ba4cffa7fe07d"} Nov 03 20:30:28.776265 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.775907 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:30:28.781755 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.781736 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:30:28.848909 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.848873 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-tls-certs\") pod \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " Nov 03 20:30:28.848909 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.848913 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-kube-api-access-kddd7\" (UniqueName: \"kubernetes.io/projected/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-kube-api-access-kddd7\") pod \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " Nov 03 20:30:28.849189 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.848979 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-dshm\") pod \"b14fd53e-9193-44a0-9693-446fe067a841\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " Nov 03 20:30:28.849189 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.849074 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-model-cache\") pod \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " Nov 03 20:30:28.849189 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.849137 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-home\" (UniqueName: \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-home\") pod \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " Nov 03 20:30:28.849365 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.849312 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b14fd53e-9193-44a0-9693-446fe067a841-tls-certs\" (UniqueName: \"kubernetes.io/secret/b14fd53e-9193-44a0-9693-446fe067a841-tls-certs\") pod \"b14fd53e-9193-44a0-9693-446fe067a841\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " Nov 03 20:30:28.849518 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.849486 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-model-cache" pod "6f5ce248-0c9e-4f39-9130-1b2aad5beb65" (UID: "6f5ce248-0c9e-4f39-9130-1b2aad5beb65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:28.849587 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.849551 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-tmp-dir\") pod \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " Nov 03 20:30:28.849668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.849648 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-model-cache\") pod \"b14fd53e-9193-44a0-9693-446fe067a841\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " Nov 03 20:30:28.849792 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.849701 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b14fd53e-9193-44a0-9693-446fe067a841-kube-api-access-zrsvz\" (UniqueName: \"kubernetes.io/projected/b14fd53e-9193-44a0-9693-446fe067a841-kube-api-access-zrsvz\") pod \"b14fd53e-9193-44a0-9693-446fe067a841\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " Nov 03 20:30:28.849792 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.849728 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-tmp-dir\") pod \"b14fd53e-9193-44a0-9693-446fe067a841\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " Nov 03 20:30:28.849922 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.849857 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-tmp-dir" pod "6f5ce248-0c9e-4f39-9130-1b2aad5beb65" (UID: "6f5ce248-0c9e-4f39-9130-1b2aad5beb65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:28.849922 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.849890 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-model-cache" pod "b14fd53e-9193-44a0-9693-446fe067a841" (UID: "b14fd53e-9193-44a0-9693-446fe067a841"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:28.850065 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.850045 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-home\" (UniqueName: \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-home\") pod \"b14fd53e-9193-44a0-9693-446fe067a841\" (UID: \"b14fd53e-9193-44a0-9693-446fe067a841\") " Nov 03 20:30:28.850131 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.850080 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-dshm\") pod \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\" (UID: \"6f5ce248-0c9e-4f39-9130-1b2aad5beb65\") " Nov 03 20:30:28.850131 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.850109 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-home" pod "6f5ce248-0c9e-4f39-9130-1b2aad5beb65" (UID: "6f5ce248-0c9e-4f39-9130-1b2aad5beb65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:28.850271 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.850150 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-tmp-dir" pod "b14fd53e-9193-44a0-9693-446fe067a841" (UID: "b14fd53e-9193-44a0-9693-446fe067a841"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:28.850326 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.850300 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-home" pod "b14fd53e-9193-44a0-9693-446fe067a841" (UID: "b14fd53e-9193-44a0-9693-446fe067a841"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:28.850973 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.850951 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-model-cache\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:28.851100 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.850975 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-home\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:28.851100 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.850989 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-tmp-dir\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:28.851100 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.851001 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-model-cache\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:28.851100 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.851014 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-tmp-dir\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:28.851100 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.851029 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-home\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:28.853962 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.853935 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-dshm" pod "b14fd53e-9193-44a0-9693-446fe067a841" (UID: "b14fd53e-9193-44a0-9693-446fe067a841"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:28.854360 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.854250 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14fd53e-9193-44a0-9693-446fe067a841-kube-api-access-zrsvz" pod "b14fd53e-9193-44a0-9693-446fe067a841" (UID: "b14fd53e-9193-44a0-9693-446fe067a841"). InnerVolumeSpecName "kube-api-access-zrsvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:28.855022 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.855001 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-tls-certs" pod "6f5ce248-0c9e-4f39-9130-1b2aad5beb65" (UID: "6f5ce248-0c9e-4f39-9130-1b2aad5beb65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:28.856778 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.856752 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-dshm" pod "6f5ce248-0c9e-4f39-9130-1b2aad5beb65" (UID: "6f5ce248-0c9e-4f39-9130-1b2aad5beb65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:28.856995 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.856972 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b14fd53e-9193-44a0-9693-446fe067a841-tls-certs" pod "b14fd53e-9193-44a0-9693-446fe067a841" (UID: "b14fd53e-9193-44a0-9693-446fe067a841"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:28.857345 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.857324 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-kube-api-access-kddd7" pod "6f5ce248-0c9e-4f39-9130-1b2aad5beb65" (UID: "6f5ce248-0c9e-4f39-9130-1b2aad5beb65"). InnerVolumeSpecName "kube-api-access-kddd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:28.953890 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.953858 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrsvz\" (UniqueName: \"kubernetes.io/projected/b14fd53e-9193-44a0-9693-446fe067a841-kube-api-access-zrsvz\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:28.953890 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.953888 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-dshm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:28.954065 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.953899 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-tls-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:28.954065 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.953908 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kddd7\" (UniqueName: \"kubernetes.io/projected/6f5ce248-0c9e-4f39-9130-1b2aad5beb65-kube-api-access-kddd7\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:28.954065 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.953917 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b14fd53e-9193-44a0-9693-446fe067a841-dshm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:28.954065 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:28.953928 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b14fd53e-9193-44a0-9693-446fe067a841-tls-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:29.758405 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:29.758368 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" Nov 03 20:30:29.758886 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:29.758400 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v" event={"ID":"6f5ce248-0c9e-4f39-9130-1b2aad5beb65","Type":"ContainerDied","Data":"0115ad0c02505ad9337dcd8e0c5e922ab41c48ecb8fe65884897a42263d19deb"} Nov 03 20:30:29.758886 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:29.758451 4918 scope.go:122] "RemoveContainer" containerID="3e4f1c5f1cac945da0445311a715a360e20769378f7435fe6c6ba4cffa7fe07d" Nov 03 20:30:29.760940 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:29.760888 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" event={"ID":"b14fd53e-9193-44a0-9693-446fe067a841","Type":"ContainerDied","Data":"be6ca1a9ff4f22217e1c507dc92cf0fec149ba6649ee5a0138726ef005b5fb29"} Nov 03 20:30:29.761073 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:29.760960 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt" Nov 03 20:30:29.771140 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:29.771116 4918 scope.go:122] "RemoveContainer" containerID="74fe3f36ca790f86d20f06f6e23b3e6a09da1850873af80fd6696dfb3b3bb417" Nov 03 20:30:29.786734 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:29.786705 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v"] Nov 03 20:30:29.792489 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:29.792466 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/distinct-w1-8013d9-kserve-5864c8c96c-qs62v"] Nov 03 20:30:29.803727 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:29.803701 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt"] Nov 03 20:30:29.809300 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:29.809281 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/premium-w1-8013d9-kserve-84847c8cf6-j2xlt"] Nov 03 20:30:31.220801 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:31.220761 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" path="/var/lib/kubelet/pods/6f5ce248-0c9e-4f39-9130-1b2aad5beb65/volumes" Nov 03 20:30:31.221324 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:31.221305 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b14fd53e-9193-44a0-9693-446fe067a841" path="/var/lib/kubelet/pods/b14fd53e-9193-44a0-9693-446fe067a841/volumes" Nov 03 20:30:33.505800 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:33.505767 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg"] Nov 03 20:30:33.506320 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:33.506025 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" podUID="f6f6c399-4702-41b4-b4f4-8920d34d7cfb" containerName="istio-proxy" containerID="cri-o://deced59ab6e154d65b4040443148f04c9d4b7283c67bf40ac3524e4053b9dce0" gracePeriod=30 Nov 03 20:30:38.803457 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.803427 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:30:38.855013 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.854973 4918 generic.go:356] "Generic (PLEG): container finished" podID="f6f6c399-4702-41b4-b4f4-8920d34d7cfb" containerID="deced59ab6e154d65b4040443148f04c9d4b7283c67bf40ac3524e4053b9dce0" exitCode=0 Nov 03 20:30:38.855347 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.855036 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" Nov 03 20:30:38.855804 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.855035 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" event={"ID":"f6f6c399-4702-41b4-b4f4-8920d34d7cfb","Type":"ContainerDied","Data":"deced59ab6e154d65b4040443148f04c9d4b7283c67bf40ac3524e4053b9dce0"} Nov 03 20:30:38.855870 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.855849 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg" event={"ID":"f6f6c399-4702-41b4-b4f4-8920d34d7cfb","Type":"ContainerDied","Data":"595576f247226938f4db15024add9d7327401ca5492fa0734a717acf060ea3db"} Nov 03 20:30:38.855912 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.855893 4918 scope.go:122] "RemoveContainer" containerID="deced59ab6e154d65b4040443148f04c9d4b7283c67bf40ac3524e4053b9dce0" Nov 03 20:30:38.896139 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.896109 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-kube-api-access-9rzgk\" (UniqueName: \"kubernetes.io/projected/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-kube-api-access-9rzgk\") pod \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " Nov 03 20:30:38.896318 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.896163 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-workload-socket\") pod \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " Nov 03 20:30:38.896318 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.896297 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-envoy\") pod \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " Nov 03 20:30:38.896431 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.896338 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-workload-certs\") pod \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " Nov 03 20:30:38.896431 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.896418 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-workload-socket" pod "f6f6c399-4702-41b4-b4f4-8920d34d7cfb" (UID: "f6f6c399-4702-41b4-b4f4-8920d34d7cfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:38.896546 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.896530 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-podinfo\") pod \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " Nov 03 20:30:38.896703 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.896650 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-workload-certs" pod "f6f6c399-4702-41b4-b4f4-8920d34d7cfb" (UID: "f6f6c399-4702-41b4-b4f4-8920d34d7cfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:38.896806 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.896711 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istiod-ca-cert\") pod \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " Nov 03 20:30:38.896875 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.896816 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-credential-socket\") pod \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " Nov 03 20:30:38.896875 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.896845 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-token\" (UniqueName: \"kubernetes.io/projected/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-token\") pod \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " Nov 03 20:30:38.896992 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.896937 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-data\") pod \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\" (UID: \"f6f6c399-4702-41b4-b4f4-8920d34d7cfb\") " Nov 03 20:30:38.898378 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.897375 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istiod-ca-cert" pod "f6f6c399-4702-41b4-b4f4-8920d34d7cfb" (UID: "f6f6c399-4702-41b4-b4f4-8920d34d7cfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:38.898378 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.897588 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-data" pod "f6f6c399-4702-41b4-b4f4-8920d34d7cfb" (UID: "f6f6c399-4702-41b4-b4f4-8920d34d7cfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:38.898378 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.897767 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-credential-socket" pod "f6f6c399-4702-41b4-b4f4-8920d34d7cfb" (UID: "f6f6c399-4702-41b4-b4f4-8920d34d7cfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:38.898378 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.898004 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:38.898378 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.898029 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:38.898378 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.898046 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:38.898378 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.898061 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:38.898378 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.898075 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:38.899135 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.899092 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-kube-api-access-9rzgk" pod "f6f6c399-4702-41b4-b4f4-8920d34d7cfb" (UID: "f6f6c399-4702-41b4-b4f4-8920d34d7cfb"). InnerVolumeSpecName "kube-api-access-9rzgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:38.900369 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.900342 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-envoy" pod "f6f6c399-4702-41b4-b4f4-8920d34d7cfb" (UID: "f6f6c399-4702-41b4-b4f4-8920d34d7cfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:38.900477 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.900451 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-token" pod "f6f6c399-4702-41b4-b4f4-8920d34d7cfb" (UID: "f6f6c399-4702-41b4-b4f4-8920d34d7cfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:38.900770 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.900750 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-podinfo" pod "f6f6c399-4702-41b4-b4f4-8920d34d7cfb" (UID: "f6f6c399-4702-41b4-b4f4-8920d34d7cfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:30:38.903045 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.903028 4918 scope.go:122] "RemoveContainer" containerID="deced59ab6e154d65b4040443148f04c9d4b7283c67bf40ac3524e4053b9dce0" Nov 03 20:30:38.903434 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:38.903407 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deced59ab6e154d65b4040443148f04c9d4b7283c67bf40ac3524e4053b9dce0\": container with ID starting with deced59ab6e154d65b4040443148f04c9d4b7283c67bf40ac3524e4053b9dce0 not found: ID does not exist" containerID="deced59ab6e154d65b4040443148f04c9d4b7283c67bf40ac3524e4053b9dce0" Nov 03 20:30:38.903516 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.903440 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deced59ab6e154d65b4040443148f04c9d4b7283c67bf40ac3524e4053b9dce0"} err="failed to get container status \"deced59ab6e154d65b4040443148f04c9d4b7283c67bf40ac3524e4053b9dce0\": rpc error: code = NotFound desc = could not find container \"deced59ab6e154d65b4040443148f04c9d4b7283c67bf40ac3524e4053b9dce0\": container with ID starting with deced59ab6e154d65b4040443148f04c9d4b7283c67bf40ac3524e4053b9dce0 not found: ID does not exist" Nov 03 20:30:38.999784 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.999750 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rzgk\" (UniqueName: \"kubernetes.io/projected/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-kube-api-access-9rzgk\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:38.999784 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.999782 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:38.999982 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.999791 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:38.999982 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:38.999801 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f6c399-4702-41b4-b4f4-8920d34d7cfb-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:39.183450 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:39.183413 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg"] Nov 03 20:30:39.187854 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:39.187814 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jlbpg"] Nov 03 20:30:39.255046 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:39.255005 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6f6c399-4702-41b4-b4f4-8920d34d7cfb" path="/var/lib/kubelet/pods/f6f6c399-4702-41b4-b4f4-8920d34d7cfb/volumes" Nov 03 20:30:44.932045 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:44.932008 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx"] Nov 03 20:30:44.932677 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:44.932248 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" podUID="74173946-29b1-4686-985c-73d7f43b51bf" containerName="maas-api" containerID="cri-o://c8fdfb9b2b2428372b82e2da4824fc6b1f270acaa6237282d8f57b2f4eb46549" gracePeriod=30 Nov 03 20:30:45.511309 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.511278 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" Nov 03 20:30:45.520035 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.520008 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs"] Nov 03 20:30:45.533695 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.533660 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8013d9-vgbhs"] Nov 03 20:30:45.576704 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.576671 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74173946-29b1-4686-985c-73d7f43b51bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74173946-29b1-4686-985c-73d7f43b51bf-maas-api-tls\") pod \"74173946-29b1-4686-985c-73d7f43b51bf\" (UID: \"74173946-29b1-4686-985c-73d7f43b51bf\") " Nov 03 20:30:45.576872 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.576723 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74173946-29b1-4686-985c-73d7f43b51bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74173946-29b1-4686-985c-73d7f43b51bf-metrics-certs\") pod \"74173946-29b1-4686-985c-73d7f43b51bf\" (UID: \"74173946-29b1-4686-985c-73d7f43b51bf\") " Nov 03 20:30:45.576952 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.576937 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74173946-29b1-4686-985c-73d7f43b51bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74173946-29b1-4686-985c-73d7f43b51bf-openshift-service-ca\") pod \"74173946-29b1-4686-985c-73d7f43b51bf\" (UID: \"74173946-29b1-4686-985c-73d7f43b51bf\") " Nov 03 20:30:45.577134 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.577117 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74173946-29b1-4686-985c-73d7f43b51bf-kube-api-access-dk6m9\" (UniqueName: \"kubernetes.io/projected/74173946-29b1-4686-985c-73d7f43b51bf-kube-api-access-dk6m9\") pod \"74173946-29b1-4686-985c-73d7f43b51bf\" (UID: \"74173946-29b1-4686-985c-73d7f43b51bf\") " Nov 03 20:30:45.577351 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.577326 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74173946-29b1-4686-985c-73d7f43b51bf-openshift-service-ca" pod "74173946-29b1-4686-985c-73d7f43b51bf" (UID: "74173946-29b1-4686-985c-73d7f43b51bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:45.577999 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.577971 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74173946-29b1-4686-985c-73d7f43b51bf-openshift-service-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:45.579384 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.579352 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74173946-29b1-4686-985c-73d7f43b51bf-maas-api-tls" pod "74173946-29b1-4686-985c-73d7f43b51bf" (UID: "74173946-29b1-4686-985c-73d7f43b51bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:45.579470 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.579399 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74173946-29b1-4686-985c-73d7f43b51bf-kube-api-access-dk6m9" pod "74173946-29b1-4686-985c-73d7f43b51bf" (UID: "74173946-29b1-4686-985c-73d7f43b51bf"). InnerVolumeSpecName "kube-api-access-dk6m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:45.579573 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.579554 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74173946-29b1-4686-985c-73d7f43b51bf-metrics-certs" pod "74173946-29b1-4686-985c-73d7f43b51bf" (UID: "74173946-29b1-4686-985c-73d7f43b51bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:45.680806 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.680766 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk6m9\" (UniqueName: \"kubernetes.io/projected/74173946-29b1-4686-985c-73d7f43b51bf-kube-api-access-dk6m9\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:45.680806 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.680806 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74173946-29b1-4686-985c-73d7f43b51bf-maas-api-tls\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:45.681096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.680820 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74173946-29b1-4686-985c-73d7f43b51bf-metrics-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:45.919941 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.919905 4918 generic.go:356] "Generic (PLEG): container finished" podID="74173946-29b1-4686-985c-73d7f43b51bf" containerID="c8fdfb9b2b2428372b82e2da4824fc6b1f270acaa6237282d8f57b2f4eb46549" exitCode=0 Nov 03 20:30:45.920116 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.919949 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" Nov 03 20:30:45.920116 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.919954 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" event={"ID":"74173946-29b1-4686-985c-73d7f43b51bf","Type":"ContainerDied","Data":"c8fdfb9b2b2428372b82e2da4824fc6b1f270acaa6237282d8f57b2f4eb46549"} Nov 03 20:30:45.920116 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.920080 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx" event={"ID":"74173946-29b1-4686-985c-73d7f43b51bf","Type":"ContainerDied","Data":"dd710af59ed0b444412857793e4974154f355f63a442d69383d810bc1a17e306"} Nov 03 20:30:45.920116 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.920110 4918 scope.go:122] "RemoveContainer" containerID="c8fdfb9b2b2428372b82e2da4824fc6b1f270acaa6237282d8f57b2f4eb46549" Nov 03 20:30:45.933267 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.933082 4918 scope.go:122] "RemoveContainer" containerID="c8fdfb9b2b2428372b82e2da4824fc6b1f270acaa6237282d8f57b2f4eb46549" Nov 03 20:30:45.933509 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:45.933355 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8fdfb9b2b2428372b82e2da4824fc6b1f270acaa6237282d8f57b2f4eb46549\": container with ID starting with c8fdfb9b2b2428372b82e2da4824fc6b1f270acaa6237282d8f57b2f4eb46549 not found: ID does not exist" containerID="c8fdfb9b2b2428372b82e2da4824fc6b1f270acaa6237282d8f57b2f4eb46549" Nov 03 20:30:45.933509 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.933381 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8fdfb9b2b2428372b82e2da4824fc6b1f270acaa6237282d8f57b2f4eb46549"} err="failed to get container status \"c8fdfb9b2b2428372b82e2da4824fc6b1f270acaa6237282d8f57b2f4eb46549\": rpc error: code = NotFound desc = could not find container \"c8fdfb9b2b2428372b82e2da4824fc6b1f270acaa6237282d8f57b2f4eb46549\": container with ID starting with c8fdfb9b2b2428372b82e2da4824fc6b1f270acaa6237282d8f57b2f4eb46549 not found: ID does not exist" Nov 03 20:30:45.947300 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.947259 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx"] Nov 03 20:30:45.953024 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:45.953000 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8013d9-9c9b99767-tt6nx"] Nov 03 20:30:47.217949 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:47.217913 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67a9be3d-a3d7-450b-926f-96fa79503381" path="/var/lib/kubelet/pods/67a9be3d-a3d7-450b-926f-96fa79503381/volumes" Nov 03 20:30:47.218630 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:47.218605 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74173946-29b1-4686-985c-73d7f43b51bf" path="/var/lib/kubelet/pods/74173946-29b1-4686-985c-73d7f43b51bf/volumes" Nov 03 20:30:48.405711 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:48.405673 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq"] Nov 03 20:30:48.406237 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:48.405919 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" podUID="11e6ea59-b2c6-409d-b2e8-4a4eec65938d" containerName="istio-proxy" containerID="cri-o://186a93d3ce6d92dcbcf17fd9fbb97331e9819264f37af51deeaca7b5f61bbb47" gracePeriod=30 Nov 03 20:30:48.427675 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:48.427646 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj"] Nov 03 20:30:48.427900 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:48.427876 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" podUID="46b614fc-7245-4556-886c-3f9118db35ad" containerName="istio-proxy" containerID="cri-o://7c99c1a48b75bbf71559dc597000e0d21065c8b2af0571db86207e4f50380bcf" gracePeriod=30 Nov 03 20:30:48.560246 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:48.558191 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7"] Nov 03 20:30:48.560246 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:48.558453 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" podUID="c1759364-4aca-4b1d-9f83-a81bc5829924" containerName="istio-proxy" containerID="cri-o://7c9a6f2352fcd5e02018c9067261dabe414841469bf1f0539ffd722c1fe5a231" gracePeriod=30 Nov 03 20:30:54.302105 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.300128 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd"] Nov 03 20:30:54.302105 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.300406 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" podUID="a7ea5a18-974d-456d-88b5-e05b8cd33f9c" containerName="istio-proxy" containerID="cri-o://f51d09be5c875de99f1b4970764e01c0c7a86249a9ec4135ad6329b200288070" gracePeriod=30 Nov 03 20:30:54.421663 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.417773 4918 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 20:30:54.421663 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.417827 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" podUID="c1759364-4aca-4b1d-9f83-a81bc5829924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 20:30:54.434045 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.424090 4918 generic.go:356] "Generic (PLEG): container finished" podID="46b614fc-7245-4556-886c-3f9118db35ad" containerID="7c99c1a48b75bbf71559dc597000e0d21065c8b2af0571db86207e4f50380bcf" exitCode=0 Nov 03 20:30:54.434045 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.424302 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" event={"ID":"46b614fc-7245-4556-886c-3f9118db35ad","Type":"ContainerDied","Data":"7c99c1a48b75bbf71559dc597000e0d21065c8b2af0571db86207e4f50380bcf"} Nov 03 20:30:54.434045 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.427576 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" podUID="97238908-155f-4656-8767-fae2a876d653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.128:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:30:54.434045 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.428642 4918 generic.go:356] "Generic (PLEG): container finished" podID="11e6ea59-b2c6-409d-b2e8-4a4eec65938d" containerID="186a93d3ce6d92dcbcf17fd9fbb97331e9819264f37af51deeaca7b5f61bbb47" exitCode=0 Nov 03 20:30:54.434045 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.429156 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" event={"ID":"11e6ea59-b2c6-409d-b2e8-4a4eec65938d","Type":"ContainerDied","Data":"186a93d3ce6d92dcbcf17fd9fbb97331e9819264f37af51deeaca7b5f61bbb47"} Nov 03 20:30:54.454721 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.454686 4918 generic.go:356] "Generic (PLEG): container finished" podID="c1759364-4aca-4b1d-9f83-a81bc5829924" containerID="7c9a6f2352fcd5e02018c9067261dabe414841469bf1f0539ffd722c1fe5a231" exitCode=0 Nov 03 20:30:54.455065 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.454872 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" event={"ID":"c1759364-4aca-4b1d-9f83-a81bc5829924","Type":"ContainerDied","Data":"7c9a6f2352fcd5e02018c9067261dabe414841469bf1f0539ffd722c1fe5a231"} Nov 03 20:30:54.647567 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.647518 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:30:54.733105 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.733051 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-istio-data\") pod \"46b614fc-7245-4556-886c-3f9118db35ad\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " Nov 03 20:30:54.733887 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.733862 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-workload-socket\") pod \"46b614fc-7245-4556-886c-3f9118db35ad\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " Nov 03 20:30:54.734475 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.734456 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-istio-envoy\") pod \"46b614fc-7245-4556-886c-3f9118db35ad\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " Nov 03 20:30:54.748030 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.748005 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-credential-socket\") pod \"46b614fc-7245-4556-886c-3f9118db35ad\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " Nov 03 20:30:54.748511 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.733686 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-istio-data" pod "46b614fc-7245-4556-886c-3f9118db35ad" (UID: "46b614fc-7245-4556-886c-3f9118db35ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.748656 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.734273 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-workload-socket" pod "46b614fc-7245-4556-886c-3f9118db35ad" (UID: "46b614fc-7245-4556-886c-3f9118db35ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.749189 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.748387 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-credential-socket" pod "46b614fc-7245-4556-886c-3f9118db35ad" (UID: "46b614fc-7245-4556-886c-3f9118db35ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.749304 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.749124 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b614fc-7245-4556-886c-3f9118db35ad-istiod-ca-cert" pod "46b614fc-7245-4556-886c-3f9118db35ad" (UID: "46b614fc-7245-4556-886c-3f9118db35ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:54.749411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.748482 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b614fc-7245-4556-886c-3f9118db35ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b614fc-7245-4556-886c-3f9118db35ad-istiod-ca-cert\") pod \"46b614fc-7245-4556-886c-3f9118db35ad\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " Nov 03 20:30:54.749602 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.749587 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b614fc-7245-4556-886c-3f9118db35ad-kube-api-access-qwzqt\" (UniqueName: \"kubernetes.io/projected/46b614fc-7245-4556-886c-3f9118db35ad-kube-api-access-qwzqt\") pod \"46b614fc-7245-4556-886c-3f9118db35ad\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " Nov 03 20:30:54.750153 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.750129 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b614fc-7245-4556-886c-3f9118db35ad-istio-token\" (UniqueName: \"kubernetes.io/projected/46b614fc-7245-4556-886c-3f9118db35ad-istio-token\") pod \"46b614fc-7245-4556-886c-3f9118db35ad\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " Nov 03 20:30:54.750694 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.750677 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46b614fc-7245-4556-886c-3f9118db35ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b614fc-7245-4556-886c-3f9118db35ad-istio-podinfo\") pod \"46b614fc-7245-4556-886c-3f9118db35ad\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " Nov 03 20:30:54.751219 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.751200 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-workload-certs\") pod \"46b614fc-7245-4556-886c-3f9118db35ad\" (UID: \"46b614fc-7245-4556-886c-3f9118db35ad\") " Nov 03 20:30:54.753615 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.753582 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:54.753762 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.753747 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:54.753882 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.753869 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:54.753977 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.753966 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b614fc-7245-4556-886c-3f9118db35ad-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:54.754055 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.751648 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-workload-certs" pod "46b614fc-7245-4556-886c-3f9118db35ad" (UID: "46b614fc-7245-4556-886c-3f9118db35ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.784461 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.784427 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b614fc-7245-4556-886c-3f9118db35ad-istio-token" pod "46b614fc-7245-4556-886c-3f9118db35ad" (UID: "46b614fc-7245-4556-886c-3f9118db35ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:54.785574 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.785443 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-istio-envoy" pod "46b614fc-7245-4556-886c-3f9118db35ad" (UID: "46b614fc-7245-4556-886c-3f9118db35ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.785845 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.785817 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b614fc-7245-4556-886c-3f9118db35ad-kube-api-access-qwzqt" pod "46b614fc-7245-4556-886c-3f9118db35ad" (UID: "46b614fc-7245-4556-886c-3f9118db35ad"). InnerVolumeSpecName "kube-api-access-qwzqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:54.796320 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.796260 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46b614fc-7245-4556-886c-3f9118db35ad-istio-podinfo" pod "46b614fc-7245-4556-886c-3f9118db35ad" (UID: "46b614fc-7245-4556-886c-3f9118db35ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:30:54.840551 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.840518 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:30:54.843890 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.843863 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:30:54.855609 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.855577 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwzqt\" (UniqueName: \"kubernetes.io/projected/46b614fc-7245-4556-886c-3f9118db35ad-kube-api-access-qwzqt\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:54.855609 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.855603 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b614fc-7245-4556-886c-3f9118db35ad-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:54.855609 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.855613 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b614fc-7245-4556-886c-3f9118db35ad-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:54.855885 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.855624 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:54.855885 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.855639 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b614fc-7245-4556-886c-3f9118db35ad-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:54.956959 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.956160 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-envoy\") pod \"c1759364-4aca-4b1d-9f83-a81bc5829924\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " Nov 03 20:30:54.956959 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.956272 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-kube-api-access-6bkzt\" (UniqueName: \"kubernetes.io/projected/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-kube-api-access-6bkzt\") pod \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " Nov 03 20:30:54.956959 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.956318 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-workload-certs\") pod \"c1759364-4aca-4b1d-9f83-a81bc5829924\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " Nov 03 20:30:54.956959 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.956364 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-data\") pod \"c1759364-4aca-4b1d-9f83-a81bc5829924\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " Nov 03 20:30:54.956959 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.956416 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-data\") pod \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " Nov 03 20:30:54.956959 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.956454 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-credential-socket\") pod \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " Nov 03 20:30:54.956959 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.956496 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1759364-4aca-4b1d-9f83-a81bc5829924-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1759364-4aca-4b1d-9f83-a81bc5829924-istiod-ca-cert\") pod \"c1759364-4aca-4b1d-9f83-a81bc5829924\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " Nov 03 20:30:54.956959 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.956588 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-workload-certs\") pod \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " Nov 03 20:30:54.956959 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.956633 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-envoy\") pod \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " Nov 03 20:30:54.956959 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.956674 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-workload-socket\") pod \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " Nov 03 20:30:54.956959 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.956791 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-workload-socket\") pod \"c1759364-4aca-4b1d-9f83-a81bc5829924\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " Nov 03 20:30:54.956959 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.956909 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-credential-socket\") pod \"c1759364-4aca-4b1d-9f83-a81bc5829924\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " Nov 03 20:30:54.958272 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.957331 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-token\" (UniqueName: \"kubernetes.io/projected/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-token\") pod \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " Nov 03 20:30:54.958272 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.957496 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1759364-4aca-4b1d-9f83-a81bc5829924-kube-api-access-vl6mr\" (UniqueName: \"kubernetes.io/projected/c1759364-4aca-4b1d-9f83-a81bc5829924-kube-api-access-vl6mr\") pod \"c1759364-4aca-4b1d-9f83-a81bc5829924\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " Nov 03 20:30:54.958272 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.957643 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-credential-socket" pod "11e6ea59-b2c6-409d-b2e8-4a4eec65938d" (UID: "11e6ea59-b2c6-409d-b2e8-4a4eec65938d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.958272 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.957647 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-podinfo\") pod \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " Nov 03 20:30:54.958272 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.957809 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-podinfo\") pod \"c1759364-4aca-4b1d-9f83-a81bc5829924\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " Nov 03 20:30:54.958272 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.957948 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-token\" (UniqueName: \"kubernetes.io/projected/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-token\") pod \"c1759364-4aca-4b1d-9f83-a81bc5829924\" (UID: \"c1759364-4aca-4b1d-9f83-a81bc5829924\") " Nov 03 20:30:54.958272 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.957983 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istiod-ca-cert\") pod \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\" (UID: \"11e6ea59-b2c6-409d-b2e8-4a4eec65938d\") " Nov 03 20:30:54.958940 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.958912 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:54.963139 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.962209 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-workload-socket" pod "11e6ea59-b2c6-409d-b2e8-4a4eec65938d" (UID: "11e6ea59-b2c6-409d-b2e8-4a4eec65938d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.963139 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.962570 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-envoy" pod "c1759364-4aca-4b1d-9f83-a81bc5829924" (UID: "c1759364-4aca-4b1d-9f83-a81bc5829924"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.963139 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.962909 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1759364-4aca-4b1d-9f83-a81bc5829924-istiod-ca-cert" pod "c1759364-4aca-4b1d-9f83-a81bc5829924" (UID: "c1759364-4aca-4b1d-9f83-a81bc5829924"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:54.963712 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.963478 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-workload-certs" pod "11e6ea59-b2c6-409d-b2e8-4a4eec65938d" (UID: "11e6ea59-b2c6-409d-b2e8-4a4eec65938d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.964445 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.964412 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istiod-ca-cert" pod "11e6ea59-b2c6-409d-b2e8-4a4eec65938d" (UID: "11e6ea59-b2c6-409d-b2e8-4a4eec65938d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:54.964850 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.964833 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-workload-certs" pod "c1759364-4aca-4b1d-9f83-a81bc5829924" (UID: "c1759364-4aca-4b1d-9f83-a81bc5829924"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.966067 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.965745 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-credential-socket" pod "c1759364-4aca-4b1d-9f83-a81bc5829924" (UID: "c1759364-4aca-4b1d-9f83-a81bc5829924"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.966067 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.965953 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-workload-socket" pod "c1759364-4aca-4b1d-9f83-a81bc5829924" (UID: "c1759364-4aca-4b1d-9f83-a81bc5829924"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.967511 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.967464 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-data" pod "c1759364-4aca-4b1d-9f83-a81bc5829924" (UID: "c1759364-4aca-4b1d-9f83-a81bc5829924"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.967511 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.967474 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-data" pod "11e6ea59-b2c6-409d-b2e8-4a4eec65938d" (UID: "11e6ea59-b2c6-409d-b2e8-4a4eec65938d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.967652 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.967556 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-kube-api-access-6bkzt" pod "11e6ea59-b2c6-409d-b2e8-4a4eec65938d" (UID: "11e6ea59-b2c6-409d-b2e8-4a4eec65938d"). InnerVolumeSpecName "kube-api-access-6bkzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:54.968056 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.968029 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-podinfo" pod "c1759364-4aca-4b1d-9f83-a81bc5829924" (UID: "c1759364-4aca-4b1d-9f83-a81bc5829924"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:30:54.969221 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.969156 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-token" pod "11e6ea59-b2c6-409d-b2e8-4a4eec65938d" (UID: "11e6ea59-b2c6-409d-b2e8-4a4eec65938d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:54.969221 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.969209 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-token" pod "c1759364-4aca-4b1d-9f83-a81bc5829924" (UID: "c1759364-4aca-4b1d-9f83-a81bc5829924"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:54.973332 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.973305 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-envoy" pod "11e6ea59-b2c6-409d-b2e8-4a4eec65938d" (UID: "11e6ea59-b2c6-409d-b2e8-4a4eec65938d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.973434 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.973409 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1759364-4aca-4b1d-9f83-a81bc5829924-kube-api-access-vl6mr" pod "c1759364-4aca-4b1d-9f83-a81bc5829924" (UID: "c1759364-4aca-4b1d-9f83-a81bc5829924"). InnerVolumeSpecName "kube-api-access-vl6mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:54.980100 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:54.980065 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-podinfo" pod "11e6ea59-b2c6-409d-b2e8-4a4eec65938d" (UID: "11e6ea59-b2c6-409d-b2e8-4a4eec65938d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:30:55.060578 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.060489 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.060578 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.060553 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.060578 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.060572 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.060878 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.060586 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl6mr\" (UniqueName: \"kubernetes.io/projected/c1759364-4aca-4b1d-9f83-a81bc5829924-kube-api-access-vl6mr\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.060878 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.060601 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.060878 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.060615 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.060878 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.060627 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.060878 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.060640 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.060878 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.060653 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.060878 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.060666 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bkzt\" (UniqueName: \"kubernetes.io/projected/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-kube-api-access-6bkzt\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.060878 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.060679 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.060878 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.060691 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1759364-4aca-4b1d-9f83-a81bc5829924-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.060878 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.060703 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.060878 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.060715 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1759364-4aca-4b1d-9f83-a81bc5829924-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.060878 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.060742 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.060878 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.060756 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.060878 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.060771 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e6ea59-b2c6-409d-b2e8-4a4eec65938d-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.472472 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.472425 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" event={"ID":"c1759364-4aca-4b1d-9f83-a81bc5829924","Type":"ContainerDied","Data":"ada96d27dc9c960cebd27a4c47e34fdfb8a2b9b3b3cc423a790e3d0ac443e273"} Nov 03 20:30:55.472990 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.472507 4918 scope.go:122] "RemoveContainer" containerID="7c9a6f2352fcd5e02018c9067261dabe414841469bf1f0539ffd722c1fe5a231" Nov 03 20:30:55.473458 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.473430 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7" Nov 03 20:30:55.478413 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.478391 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" Nov 03 20:30:55.478717 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.478424 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj" event={"ID":"46b614fc-7245-4556-886c-3f9118db35ad","Type":"ContainerDied","Data":"6660614056afc390353270d1ab3eb20ba817a4371753a46e5e56d2397671f64f"} Nov 03 20:30:55.480347 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.480323 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" event={"ID":"11e6ea59-b2c6-409d-b2e8-4a4eec65938d","Type":"ContainerDied","Data":"b75caeefb166dc571500b68351b7d3a70c8e1a102733027fdbf43a5d6d0cd785"} Nov 03 20:30:55.480448 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.480354 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq" Nov 03 20:30:55.489604 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.489584 4918 scope.go:122] "RemoveContainer" containerID="7c99c1a48b75bbf71559dc597000e0d21065c8b2af0571db86207e4f50380bcf" Nov 03 20:30:55.501904 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.501880 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7"] Nov 03 20:30:55.504709 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.504686 4918 scope.go:122] "RemoveContainer" containerID="186a93d3ce6d92dcbcf17fd9fbb97331e9819264f37af51deeaca7b5f61bbb47" Nov 03 20:30:55.512242 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.512212 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-xfpw7"] Nov 03 20:30:55.530656 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.530622 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj"] Nov 03 20:30:55.551070 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.551040 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w2tcj"] Nov 03 20:30:55.558639 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.558609 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq"] Nov 03 20:30:55.563537 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:55.563512 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-sg6vq"] Nov 03 20:30:57.218440 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:57.218407 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11e6ea59-b2c6-409d-b2e8-4a4eec65938d" path="/var/lib/kubelet/pods/11e6ea59-b2c6-409d-b2e8-4a4eec65938d/volumes" Nov 03 20:30:57.219054 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:57.219036 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b614fc-7245-4556-886c-3f9118db35ad" path="/var/lib/kubelet/pods/46b614fc-7245-4556-886c-3f9118db35ad/volumes" Nov 03 20:30:57.219621 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:57.219606 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1759364-4aca-4b1d-9f83-a81bc5829924" path="/var/lib/kubelet/pods/c1759364-4aca-4b1d-9f83-a81bc5829924/volumes" Nov 03 20:30:58.080922 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:58.080885 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm"] Nov 03 20:30:58.081159 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:30:58.081132 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" podUID="6dad3fad-396e-4647-a9bf-a7a906fdd1fe" containerName="istio-proxy" containerID="cri-o://ab1aca5343b895af833d4d40863d1d20d4d364a3edae815679eac4e90e777704" gracePeriod=30 Nov 03 20:30:58.221153 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:30:58.221112 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:31:00.112556 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.112522 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:31:00.199369 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.199277 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istiod-ca-cert\") pod \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " Nov 03 20:31:00.199369 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.199351 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-podinfo\") pod \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " Nov 03 20:31:00.199583 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.199531 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-data\") pod \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " Nov 03 20:31:00.199692 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.199669 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-credential-socket\") pod \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " Nov 03 20:31:00.199794 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.199775 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-token\" (UniqueName: \"kubernetes.io/projected/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-token\") pod \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " Nov 03 20:31:00.199794 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.199704 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istiod-ca-cert" pod "a7ea5a18-974d-456d-88b5-e05b8cd33f9c" (UID: "a7ea5a18-974d-456d-88b5-e05b8cd33f9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:00.199914 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.199772 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-data" pod "a7ea5a18-974d-456d-88b5-e05b8cd33f9c" (UID: "a7ea5a18-974d-456d-88b5-e05b8cd33f9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:00.199914 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.199865 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-envoy\") pod \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " Nov 03 20:31:00.200023 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.199922 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-kube-api-access-qb7k4\" (UniqueName: \"kubernetes.io/projected/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-kube-api-access-qb7k4\") pod \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " Nov 03 20:31:00.200023 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.199956 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-workload-socket\") pod \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " Nov 03 20:31:00.200023 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.199964 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-credential-socket" pod "a7ea5a18-974d-456d-88b5-e05b8cd33f9c" (UID: "a7ea5a18-974d-456d-88b5-e05b8cd33f9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:00.200214 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.200036 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-workload-certs\") pod \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\" (UID: \"a7ea5a18-974d-456d-88b5-e05b8cd33f9c\") " Nov 03 20:31:00.200548 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.200508 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-workload-socket" pod "a7ea5a18-974d-456d-88b5-e05b8cd33f9c" (UID: "a7ea5a18-974d-456d-88b5-e05b8cd33f9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:00.200763 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.200734 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-workload-certs" pod "a7ea5a18-974d-456d-88b5-e05b8cd33f9c" (UID: "a7ea5a18-974d-456d-88b5-e05b8cd33f9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:00.201139 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.201117 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.201280 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.201144 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.201280 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.201160 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.201280 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.201191 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.201280 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.201207 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.201893 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.201871 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-podinfo" pod "a7ea5a18-974d-456d-88b5-e05b8cd33f9c" (UID: "a7ea5a18-974d-456d-88b5-e05b8cd33f9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:31:00.202265 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.202246 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-token" pod "a7ea5a18-974d-456d-88b5-e05b8cd33f9c" (UID: "a7ea5a18-974d-456d-88b5-e05b8cd33f9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:00.202542 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.202526 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-kube-api-access-qb7k4" pod "a7ea5a18-974d-456d-88b5-e05b8cd33f9c" (UID: "a7ea5a18-974d-456d-88b5-e05b8cd33f9c"). InnerVolumeSpecName "kube-api-access-qb7k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:00.202628 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.202616 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-envoy" pod "a7ea5a18-974d-456d-88b5-e05b8cd33f9c" (UID: "a7ea5a18-974d-456d-88b5-e05b8cd33f9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:00.303507 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.303402 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.303507 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.303440 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.303507 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.303455 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb7k4\" (UniqueName: \"kubernetes.io/projected/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-kube-api-access-qb7k4\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.303507 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.303471 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ea5a18-974d-456d-88b5-e05b8cd33f9c-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.568930 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.568895 4918 generic.go:356] "Generic (PLEG): container finished" podID="a7ea5a18-974d-456d-88b5-e05b8cd33f9c" containerID="f51d09be5c875de99f1b4970764e01c0c7a86249a9ec4135ad6329b200288070" exitCode=0 Nov 03 20:31:00.569102 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.568945 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" Nov 03 20:31:00.569102 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.568974 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" event={"ID":"a7ea5a18-974d-456d-88b5-e05b8cd33f9c","Type":"ContainerDied","Data":"f51d09be5c875de99f1b4970764e01c0c7a86249a9ec4135ad6329b200288070"} Nov 03 20:31:00.569102 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.569038 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd" event={"ID":"a7ea5a18-974d-456d-88b5-e05b8cd33f9c","Type":"ContainerDied","Data":"665f0fe3977d192028484f5dd4ae59c02e09fb02154363917784769a528e0dce"} Nov 03 20:31:00.569102 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.569076 4918 scope.go:122] "RemoveContainer" containerID="f51d09be5c875de99f1b4970764e01c0c7a86249a9ec4135ad6329b200288070" Nov 03 20:31:00.581767 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.581750 4918 scope.go:122] "RemoveContainer" containerID="f51d09be5c875de99f1b4970764e01c0c7a86249a9ec4135ad6329b200288070" Nov 03 20:31:00.582032 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:31:00.582012 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f51d09be5c875de99f1b4970764e01c0c7a86249a9ec4135ad6329b200288070\": container with ID starting with f51d09be5c875de99f1b4970764e01c0c7a86249a9ec4135ad6329b200288070 not found: ID does not exist" containerID="f51d09be5c875de99f1b4970764e01c0c7a86249a9ec4135ad6329b200288070" Nov 03 20:31:00.582076 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.582039 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51d09be5c875de99f1b4970764e01c0c7a86249a9ec4135ad6329b200288070"} err="failed to get container status \"f51d09be5c875de99f1b4970764e01c0c7a86249a9ec4135ad6329b200288070\": rpc error: code = NotFound desc = could not find container \"f51d09be5c875de99f1b4970764e01c0c7a86249a9ec4135ad6329b200288070\": container with ID starting with f51d09be5c875de99f1b4970764e01c0c7a86249a9ec4135ad6329b200288070 not found: ID does not exist" Nov 03 20:31:00.603389 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.603357 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd"] Nov 03 20:31:00.610045 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:00.610013 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g2tzd"] Nov 03 20:31:01.217618 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:01.217585 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7ea5a18-974d-456d-88b5-e05b8cd33f9c" path="/var/lib/kubelet/pods/a7ea5a18-974d-456d-88b5-e05b8cd33f9c/volumes" Nov 03 20:31:03.376206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.376165 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:31:03.481738 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.481699 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-workload-certs\") pod \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " Nov 03 20:31:03.481947 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.481800 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-credential-socket\") pod \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " Nov 03 20:31:03.481947 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.481887 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istiod-ca-cert\") pod \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " Nov 03 20:31:03.482078 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.482008 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-workload-certs" pod "6dad3fad-396e-4647-a9bf-a7a906fdd1fe" (UID: "6dad3fad-396e-4647-a9bf-a7a906fdd1fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:03.482078 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.482025 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-data\") pod \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " Nov 03 20:31:03.482217 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.482103 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-credential-socket" pod "6dad3fad-396e-4647-a9bf-a7a906fdd1fe" (UID: "6dad3fad-396e-4647-a9bf-a7a906fdd1fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:03.482217 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.482126 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-token\" (UniqueName: \"kubernetes.io/projected/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-token\") pod \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " Nov 03 20:31:03.482337 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.482255 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-data" pod "6dad3fad-396e-4647-a9bf-a7a906fdd1fe" (UID: "6dad3fad-396e-4647-a9bf-a7a906fdd1fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:03.482337 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.482274 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-podinfo\") pod \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " Nov 03 20:31:03.482337 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.482299 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istiod-ca-cert" pod "6dad3fad-396e-4647-a9bf-a7a906fdd1fe" (UID: "6dad3fad-396e-4647-a9bf-a7a906fdd1fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:03.482337 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.482320 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-workload-socket\") pod \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " Nov 03 20:31:03.482546 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.482356 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-kube-api-access-9kgwq\" (UniqueName: \"kubernetes.io/projected/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-kube-api-access-9kgwq\") pod \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " Nov 03 20:31:03.482546 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.482395 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-envoy\") pod \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\" (UID: \"6dad3fad-396e-4647-a9bf-a7a906fdd1fe\") " Nov 03 20:31:03.482661 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.482586 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-workload-socket" pod "6dad3fad-396e-4647-a9bf-a7a906fdd1fe" (UID: "6dad3fad-396e-4647-a9bf-a7a906fdd1fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:03.483518 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.483484 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:03.483697 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.483531 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:03.483697 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.483557 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:03.483697 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.483577 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:03.483697 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.483602 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:03.484683 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.484653 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-token" pod "6dad3fad-396e-4647-a9bf-a7a906fdd1fe" (UID: "6dad3fad-396e-4647-a9bf-a7a906fdd1fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:03.485038 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.485016 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-podinfo" pod "6dad3fad-396e-4647-a9bf-a7a906fdd1fe" (UID: "6dad3fad-396e-4647-a9bf-a7a906fdd1fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:31:03.485386 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.485157 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-kube-api-access-9kgwq" pod "6dad3fad-396e-4647-a9bf-a7a906fdd1fe" (UID: "6dad3fad-396e-4647-a9bf-a7a906fdd1fe"). InnerVolumeSpecName "kube-api-access-9kgwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:03.486490 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.486222 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-envoy" pod "6dad3fad-396e-4647-a9bf-a7a906fdd1fe" (UID: "6dad3fad-396e-4647-a9bf-a7a906fdd1fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:03.587206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.586291 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:03.587206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.586338 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:03.587206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.586358 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kgwq\" (UniqueName: \"kubernetes.io/projected/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-kube-api-access-9kgwq\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:03.587206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.586381 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dad3fad-396e-4647-a9bf-a7a906fdd1fe-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:03.605404 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.605295 4918 generic.go:356] "Generic (PLEG): container finished" podID="6dad3fad-396e-4647-a9bf-a7a906fdd1fe" containerID="ab1aca5343b895af833d4d40863d1d20d4d364a3edae815679eac4e90e777704" exitCode=0 Nov 03 20:31:03.605641 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.605441 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" Nov 03 20:31:03.605641 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.605526 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" event={"ID":"6dad3fad-396e-4647-a9bf-a7a906fdd1fe","Type":"ContainerDied","Data":"ab1aca5343b895af833d4d40863d1d20d4d364a3edae815679eac4e90e777704"} Nov 03 20:31:03.605641 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.605595 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm" event={"ID":"6dad3fad-396e-4647-a9bf-a7a906fdd1fe","Type":"ContainerDied","Data":"c34953b9bc5f8207ffe05156a14dffb9cec55e989cf6dc2354aa5894c09d0128"} Nov 03 20:31:03.605641 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.605618 4918 scope.go:122] "RemoveContainer" containerID="ab1aca5343b895af833d4d40863d1d20d4d364a3edae815679eac4e90e777704" Nov 03 20:31:03.624471 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.624419 4918 scope.go:122] "RemoveContainer" containerID="ab1aca5343b895af833d4d40863d1d20d4d364a3edae815679eac4e90e777704" Nov 03 20:31:03.624787 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:31:03.624760 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab1aca5343b895af833d4d40863d1d20d4d364a3edae815679eac4e90e777704\": container with ID starting with ab1aca5343b895af833d4d40863d1d20d4d364a3edae815679eac4e90e777704 not found: ID does not exist" containerID="ab1aca5343b895af833d4d40863d1d20d4d364a3edae815679eac4e90e777704" Nov 03 20:31:03.624935 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.624798 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab1aca5343b895af833d4d40863d1d20d4d364a3edae815679eac4e90e777704"} err="failed to get container status \"ab1aca5343b895af833d4d40863d1d20d4d364a3edae815679eac4e90e777704\": rpc error: code = NotFound desc = could not find container \"ab1aca5343b895af833d4d40863d1d20d4d364a3edae815679eac4e90e777704\": container with ID starting with ab1aca5343b895af833d4d40863d1d20d4d364a3edae815679eac4e90e777704 not found: ID does not exist" Nov 03 20:31:03.635281 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.635248 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm"] Nov 03 20:31:03.639306 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:03.639277 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-czttm"] Nov 03 20:31:05.218134 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:05.218097 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dad3fad-396e-4647-a9bf-a7a906fdd1fe" path="/var/lib/kubelet/pods/6dad3fad-396e-4647-a9bf-a7a906fdd1fe/volumes" Nov 03 20:31:28.210763 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:31:28.210726 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:31:30.061958 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:30.061924 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4"] Nov 03 20:31:45.085511 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:45.085472 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" podUID="df7b9bb4-44fb-4272-a6c7-90e71d051f46" containerName="main" containerID="cri-o://2ca781493b1a6fb857f62b1617c5270af94dbd7d42a6a7c12bcf80b8d15941be" gracePeriod=45 Nov 03 20:31:45.389941 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:45.389916 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:31:45.458953 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:45.458917 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-dshm\") pod \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " Nov 03 20:31:45.459149 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:45.459016 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df7b9bb4-44fb-4272-a6c7-90e71d051f46-kube-api-access-bdvbr\" (UniqueName: \"kubernetes.io/projected/df7b9bb4-44fb-4272-a6c7-90e71d051f46-kube-api-access-bdvbr\") pod \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " Nov 03 20:31:45.459149 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:45.459081 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-home\" (UniqueName: \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-home\") pod \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " Nov 03 20:31:45.459149 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:45.459110 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df7b9bb4-44fb-4272-a6c7-90e71d051f46-tls-certs\" (UniqueName: \"kubernetes.io/secret/df7b9bb4-44fb-4272-a6c7-90e71d051f46-tls-certs\") pod \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " Nov 03 20:31:45.459368 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:45.459336 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-tmp-dir\") pod \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " Nov 03 20:31:45.459424 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:45.459384 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-model-cache\") pod \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\" (UID: \"df7b9bb4-44fb-4272-a6c7-90e71d051f46\") " Nov 03 20:31:45.459797 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:45.459770 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-home" pod "df7b9bb4-44fb-4272-a6c7-90e71d051f46" (UID: "df7b9bb4-44fb-4272-a6c7-90e71d051f46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:45.460412 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:45.460091 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-model-cache" pod "df7b9bb4-44fb-4272-a6c7-90e71d051f46" (UID: "df7b9bb4-44fb-4272-a6c7-90e71d051f46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:45.460767 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:45.460745 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-tmp-dir" pod "df7b9bb4-44fb-4272-a6c7-90e71d051f46" (UID: "df7b9bb4-44fb-4272-a6c7-90e71d051f46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:45.463152 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:45.463122 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-home\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:45.463152 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:45.463151 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-tmp-dir\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:45.463387 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:45.463169 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-model-cache\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:45.463956 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:45.463921 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df7b9bb4-44fb-4272-a6c7-90e71d051f46-tls-certs" pod "df7b9bb4-44fb-4272-a6c7-90e71d051f46" (UID: "df7b9bb4-44fb-4272-a6c7-90e71d051f46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:45.465614 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:45.465444 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-dshm" pod "df7b9bb4-44fb-4272-a6c7-90e71d051f46" (UID: "df7b9bb4-44fb-4272-a6c7-90e71d051f46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:45.465895 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:45.465866 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7b9bb4-44fb-4272-a6c7-90e71d051f46-kube-api-access-bdvbr" pod "df7b9bb4-44fb-4272-a6c7-90e71d051f46" (UID: "df7b9bb4-44fb-4272-a6c7-90e71d051f46"). InnerVolumeSpecName "kube-api-access-bdvbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:45.568867 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:45.565384 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7b9bb4-44fb-4272-a6c7-90e71d051f46-dshm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:45.568867 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:45.565428 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdvbr\" (UniqueName: \"kubernetes.io/projected/df7b9bb4-44fb-4272-a6c7-90e71d051f46-kube-api-access-bdvbr\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:45.568867 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:45.565446 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df7b9bb4-44fb-4272-a6c7-90e71d051f46-tls-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:46.029040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:46.029008 4918 generic.go:356] "Generic (PLEG): container finished" podID="df7b9bb4-44fb-4272-a6c7-90e71d051f46" containerID="2ca781493b1a6fb857f62b1617c5270af94dbd7d42a6a7c12bcf80b8d15941be" exitCode=0 Nov 03 20:31:46.029234 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:46.029056 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" Nov 03 20:31:46.029234 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:46.029088 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" event={"ID":"df7b9bb4-44fb-4272-a6c7-90e71d051f46","Type":"ContainerDied","Data":"2ca781493b1a6fb857f62b1617c5270af94dbd7d42a6a7c12bcf80b8d15941be"} Nov 03 20:31:46.029234 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:46.029139 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4" event={"ID":"df7b9bb4-44fb-4272-a6c7-90e71d051f46","Type":"ContainerDied","Data":"57e892041478ce21ef9e0bc0cfb4938aae3d753332997c754e9d76b1ac024a44"} Nov 03 20:31:46.029234 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:46.029162 4918 scope.go:122] "RemoveContainer" containerID="2ca781493b1a6fb857f62b1617c5270af94dbd7d42a6a7c12bcf80b8d15941be" Nov 03 20:31:46.041645 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:46.041620 4918 scope.go:122] "RemoveContainer" containerID="2ca781493b1a6fb857f62b1617c5270af94dbd7d42a6a7c12bcf80b8d15941be" Nov 03 20:31:46.041909 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:31:46.041889 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca781493b1a6fb857f62b1617c5270af94dbd7d42a6a7c12bcf80b8d15941be\": container with ID starting with 2ca781493b1a6fb857f62b1617c5270af94dbd7d42a6a7c12bcf80b8d15941be not found: ID does not exist" containerID="2ca781493b1a6fb857f62b1617c5270af94dbd7d42a6a7c12bcf80b8d15941be" Nov 03 20:31:46.041963 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:46.041916 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca781493b1a6fb857f62b1617c5270af94dbd7d42a6a7c12bcf80b8d15941be"} err="failed to get container status \"2ca781493b1a6fb857f62b1617c5270af94dbd7d42a6a7c12bcf80b8d15941be\": rpc error: code = NotFound desc = could not find container \"2ca781493b1a6fb857f62b1617c5270af94dbd7d42a6a7c12bcf80b8d15941be\": container with ID starting with 2ca781493b1a6fb857f62b1617c5270af94dbd7d42a6a7c12bcf80b8d15941be not found: ID does not exist" Nov 03 20:31:46.109435 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:46.109396 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4"] Nov 03 20:31:46.117291 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:46.117258 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-77e83b/test-model-77e83b-kserve-78ddf686b7-wr9b4"] Nov 03 20:31:47.280201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:47.279332 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df7b9bb4-44fb-4272-a6c7-90e71d051f46" path="/var/lib/kubelet/pods/df7b9bb4-44fb-4272-a6c7-90e71d051f46/volumes" Nov 03 20:31:57.527014 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:57.526984 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:31:57.534434 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:31:57.534405 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:31:58.209905 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:31:58.209868 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:32:20.295360 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:20.295280 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd"] Nov 03 20:32:28.211066 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:32:28.211031 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:32:30.589158 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:30.589119 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2"] Nov 03 20:32:35.317117 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.317051 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" podUID="8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0" containerName="main" containerID="cri-o://489e10e04f01626abd2903685fd26fcb7353ac44183c963f337d9bb4e01a47ce" gracePeriod=45 Nov 03 20:32:35.576162 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.576077 4918 generic.go:356] "Generic (PLEG): container finished" podID="8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0" containerID="489e10e04f01626abd2903685fd26fcb7353ac44183c963f337d9bb4e01a47ce" exitCode=0 Nov 03 20:32:35.576162 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.576139 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" event={"ID":"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0","Type":"ContainerDied","Data":"489e10e04f01626abd2903685fd26fcb7353ac44183c963f337d9bb4e01a47ce"} Nov 03 20:32:35.657270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.657242 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:32:35.782283 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.782251 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-tls-certs\" (UniqueName: \"kubernetes.io/secret/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-tls-certs\") pod \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " Nov 03 20:32:35.782283 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.782292 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-tmp-dir\") pod \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " Nov 03 20:32:35.782515 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.782389 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-dshm\") pod \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " Nov 03 20:32:35.782515 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.782437 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-kube-api-access-xp8q9\" (UniqueName: \"kubernetes.io/projected/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-kube-api-access-xp8q9\") pod \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " Nov 03 20:32:35.782625 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.782580 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-tmp-dir" pod "8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0" (UID: "8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:35.782686 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.782625 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-home\" (UniqueName: \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-home\") pod \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " Nov 03 20:32:35.782686 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.782665 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-model-cache\") pod \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\" (UID: \"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0\") " Nov 03 20:32:35.783129 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.783053 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-model-cache" pod "8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0" (UID: "8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:35.783615 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.783587 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-model-cache\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.783728 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.783619 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-tmp-dir\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.783728 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.783695 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-home" pod "8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0" (UID: "8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:35.784792 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.784757 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-tls-certs" pod "8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0" (UID: "8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:35.784792 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.784777 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-kube-api-access-xp8q9" pod "8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0" (UID: "8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0"). InnerVolumeSpecName "kube-api-access-xp8q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:35.784958 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.784807 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-dshm" pod "8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0" (UID: "8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:35.885157 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.885046 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-dshm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.885157 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.885080 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp8q9\" (UniqueName: \"kubernetes.io/projected/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-kube-api-access-xp8q9\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.885157 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.885092 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-home\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.885157 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:35.885107 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0-tls-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:36.587633 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:36.587598 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" Nov 03 20:32:36.587633 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:36.587611 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd" event={"ID":"8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0","Type":"ContainerDied","Data":"8cf5b1a6e3e008be90db1e61ea2f351fd0615af616ec55dead297db2bd79d0a8"} Nov 03 20:32:36.588116 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:36.587656 4918 scope.go:122] "RemoveContainer" containerID="489e10e04f01626abd2903685fd26fcb7353ac44183c963f337d9bb4e01a47ce" Nov 03 20:32:36.614891 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:36.614852 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd"] Nov 03 20:32:36.618764 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:36.618729 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6cf8ae/test-model-6cf8ae-kserve-66b5ddb8b9-pzlkd"] Nov 03 20:32:37.219040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:37.219001 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0" path="/var/lib/kubelet/pods/8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0/volumes" Nov 03 20:32:38.108115 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:38.108078 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt"] Nov 03 20:32:41.820223 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:41.820187 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92"] Nov 03 20:32:45.637571 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:45.637523 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" podUID="a842122e-88c2-4a73-aabf-727008e66cd0" containerName="main" containerID="cri-o://1d41b1e7ba7a3a17a0b7b7bd6d275e32f2733d3de3da25fea19de6bb9ee3de8f" gracePeriod=45 Nov 03 20:32:45.991075 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:45.991041 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:32:46.010953 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.010922 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a842122e-88c2-4a73-aabf-727008e66cd0-kube-api-access-thf76\" (UniqueName: \"kubernetes.io/projected/a842122e-88c2-4a73-aabf-727008e66cd0-kube-api-access-thf76\") pod \"a842122e-88c2-4a73-aabf-727008e66cd0\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " Nov 03 20:32:46.011122 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.010987 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-dshm\") pod \"a842122e-88c2-4a73-aabf-727008e66cd0\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " Nov 03 20:32:46.011198 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.011117 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-model-cache\") pod \"a842122e-88c2-4a73-aabf-727008e66cd0\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " Nov 03 20:32:46.011760 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.011715 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-model-cache" pod "a842122e-88c2-4a73-aabf-727008e66cd0" (UID: "a842122e-88c2-4a73-aabf-727008e66cd0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:46.014352 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.014326 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-tmp-dir\") pod \"a842122e-88c2-4a73-aabf-727008e66cd0\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " Nov 03 20:32:46.014580 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.014554 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-tmp-dir" pod "a842122e-88c2-4a73-aabf-727008e66cd0" (UID: "a842122e-88c2-4a73-aabf-727008e66cd0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:46.014674 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.014621 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-home\" (UniqueName: \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-home\") pod \"a842122e-88c2-4a73-aabf-727008e66cd0\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " Nov 03 20:32:46.014736 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.014708 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a842122e-88c2-4a73-aabf-727008e66cd0-tls-certs\" (UniqueName: \"kubernetes.io/secret/a842122e-88c2-4a73-aabf-727008e66cd0-tls-certs\") pod \"a842122e-88c2-4a73-aabf-727008e66cd0\" (UID: \"a842122e-88c2-4a73-aabf-727008e66cd0\") " Nov 03 20:32:46.014861 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.014837 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-home" pod "a842122e-88c2-4a73-aabf-727008e66cd0" (UID: "a842122e-88c2-4a73-aabf-727008e66cd0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:46.015429 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.015405 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-model-cache\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:46.015573 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.015557 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-tmp-dir\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:46.015675 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.015662 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-home\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:46.018742 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.018716 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-dshm" pod "a842122e-88c2-4a73-aabf-727008e66cd0" (UID: "a842122e-88c2-4a73-aabf-727008e66cd0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:46.019084 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.019061 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a842122e-88c2-4a73-aabf-727008e66cd0-tls-certs" pod "a842122e-88c2-4a73-aabf-727008e66cd0" (UID: "a842122e-88c2-4a73-aabf-727008e66cd0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:46.019363 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.019338 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a842122e-88c2-4a73-aabf-727008e66cd0-kube-api-access-thf76" pod "a842122e-88c2-4a73-aabf-727008e66cd0" (UID: "a842122e-88c2-4a73-aabf-727008e66cd0"). InnerVolumeSpecName "kube-api-access-thf76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:46.117668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.117634 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a842122e-88c2-4a73-aabf-727008e66cd0-tls-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:46.117668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.117666 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thf76\" (UniqueName: \"kubernetes.io/projected/a842122e-88c2-4a73-aabf-727008e66cd0-kube-api-access-thf76\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:46.117668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.117676 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a842122e-88c2-4a73-aabf-727008e66cd0-dshm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:46.682783 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.682747 4918 generic.go:356] "Generic (PLEG): container finished" podID="a842122e-88c2-4a73-aabf-727008e66cd0" containerID="1d41b1e7ba7a3a17a0b7b7bd6d275e32f2733d3de3da25fea19de6bb9ee3de8f" exitCode=0 Nov 03 20:32:46.683241 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.682802 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" Nov 03 20:32:46.683241 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.682831 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" event={"ID":"a842122e-88c2-4a73-aabf-727008e66cd0","Type":"ContainerDied","Data":"1d41b1e7ba7a3a17a0b7b7bd6d275e32f2733d3de3da25fea19de6bb9ee3de8f"} Nov 03 20:32:46.683241 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.682879 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" event={"ID":"a842122e-88c2-4a73-aabf-727008e66cd0","Type":"ContainerDied","Data":"1e4a1d4e7ad0e1ff6a758b027ffdd3bf897286a11c4fbd7445063eba175afe4f"} Nov 03 20:32:46.683241 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.682900 4918 scope.go:122] "RemoveContainer" containerID="1d41b1e7ba7a3a17a0b7b7bd6d275e32f2733d3de3da25fea19de6bb9ee3de8f" Nov 03 20:32:46.697962 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.697938 4918 scope.go:122] "RemoveContainer" containerID="1d41b1e7ba7a3a17a0b7b7bd6d275e32f2733d3de3da25fea19de6bb9ee3de8f" Nov 03 20:32:46.698331 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:32:46.698308 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d41b1e7ba7a3a17a0b7b7bd6d275e32f2733d3de3da25fea19de6bb9ee3de8f\": container with ID starting with 1d41b1e7ba7a3a17a0b7b7bd6d275e32f2733d3de3da25fea19de6bb9ee3de8f not found: ID does not exist" containerID="1d41b1e7ba7a3a17a0b7b7bd6d275e32f2733d3de3da25fea19de6bb9ee3de8f" Nov 03 20:32:46.698396 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.698341 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d41b1e7ba7a3a17a0b7b7bd6d275e32f2733d3de3da25fea19de6bb9ee3de8f"} err="failed to get container status \"1d41b1e7ba7a3a17a0b7b7bd6d275e32f2733d3de3da25fea19de6bb9ee3de8f\": rpc error: code = NotFound desc = could not find container \"1d41b1e7ba7a3a17a0b7b7bd6d275e32f2733d3de3da25fea19de6bb9ee3de8f\": container with ID starting with 1d41b1e7ba7a3a17a0b7b7bd6d275e32f2733d3de3da25fea19de6bb9ee3de8f not found: ID does not exist" Nov 03 20:32:46.710334 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.710307 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2"] Nov 03 20:32:46.715797 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.715769 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2"] Nov 03 20:32:46.963901 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:46.963816 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8c684d/test-model-8c684d-kserve-6899d979d5-gpqr2" podUID="a842122e-88c2-4a73-aabf-727008e66cd0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.143:8000/ready\": context deadline exceeded" Nov 03 20:32:47.218113 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:47.218035 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a842122e-88c2-4a73-aabf-727008e66cd0" path="/var/lib/kubelet/pods/a842122e-88c2-4a73-aabf-727008e66cd0/volumes" Nov 03 20:32:53.129741 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.129699 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" podUID="97238908-155f-4656-8767-fae2a876d653" containerName="main" containerID="cri-o://d6b8b9d717059d83f58b8196843dde345caf04f005df518a62bae98cb105637d" gracePeriod=45 Nov 03 20:32:53.411512 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.411482 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:32:53.422660 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.422628 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-dshm\") pod \"97238908-155f-4656-8767-fae2a876d653\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " Nov 03 20:32:53.422802 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.422726 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97238908-155f-4656-8767-fae2a876d653-tls-certs\" (UniqueName: \"kubernetes.io/secret/97238908-155f-4656-8767-fae2a876d653-tls-certs\") pod \"97238908-155f-4656-8767-fae2a876d653\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " Nov 03 20:32:53.422880 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.422808 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-tmp-dir\") pod \"97238908-155f-4656-8767-fae2a876d653\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " Nov 03 20:32:53.422880 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.422846 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97238908-155f-4656-8767-fae2a876d653-kube-api-access-zxshx\" (UniqueName: \"kubernetes.io/projected/97238908-155f-4656-8767-fae2a876d653-kube-api-access-zxshx\") pod \"97238908-155f-4656-8767-fae2a876d653\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " Nov 03 20:32:53.422997 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.422887 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-home\" (UniqueName: \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-home\") pod \"97238908-155f-4656-8767-fae2a876d653\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " Nov 03 20:32:53.422997 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.422949 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-model-cache\") pod \"97238908-155f-4656-8767-fae2a876d653\" (UID: \"97238908-155f-4656-8767-fae2a876d653\") " Nov 03 20:32:53.423118 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.423077 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-tmp-dir" pod "97238908-155f-4656-8767-fae2a876d653" (UID: "97238908-155f-4656-8767-fae2a876d653"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:53.423172 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.423148 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-home" pod "97238908-155f-4656-8767-fae2a876d653" (UID: "97238908-155f-4656-8767-fae2a876d653"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:53.423342 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.423317 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-model-cache" pod "97238908-155f-4656-8767-fae2a876d653" (UID: "97238908-155f-4656-8767-fae2a876d653"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:53.423697 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.423676 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-tmp-dir\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:53.423697 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.423696 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-home\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:53.423820 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.423704 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-model-cache\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:53.424905 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.424886 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97238908-155f-4656-8767-fae2a876d653-tls-certs" pod "97238908-155f-4656-8767-fae2a876d653" (UID: "97238908-155f-4656-8767-fae2a876d653"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:53.425170 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.425152 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-dshm" pod "97238908-155f-4656-8767-fae2a876d653" (UID: "97238908-155f-4656-8767-fae2a876d653"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:53.425266 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.425242 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97238908-155f-4656-8767-fae2a876d653-kube-api-access-zxshx" pod "97238908-155f-4656-8767-fae2a876d653" (UID: "97238908-155f-4656-8767-fae2a876d653"). InnerVolumeSpecName "kube-api-access-zxshx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:53.525438 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.525404 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97238908-155f-4656-8767-fae2a876d653-dshm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:53.525438 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.525434 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97238908-155f-4656-8767-fae2a876d653-tls-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:53.525639 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.525447 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxshx\" (UniqueName: \"kubernetes.io/projected/97238908-155f-4656-8767-fae2a876d653-kube-api-access-zxshx\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:53.800662 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.800554 4918 generic.go:356] "Generic (PLEG): container finished" podID="97238908-155f-4656-8767-fae2a876d653" containerID="d6b8b9d717059d83f58b8196843dde345caf04f005df518a62bae98cb105637d" exitCode=0 Nov 03 20:32:53.800662 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.800630 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" Nov 03 20:32:53.800900 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.800651 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" event={"ID":"97238908-155f-4656-8767-fae2a876d653","Type":"ContainerDied","Data":"d6b8b9d717059d83f58b8196843dde345caf04f005df518a62bae98cb105637d"} Nov 03 20:32:53.800900 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.800709 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" event={"ID":"97238908-155f-4656-8767-fae2a876d653","Type":"ContainerDied","Data":"1e6c26049fd6dfdd9a2350437c0e9cc78989b7242cb5a7be0b7838f4a6142f8c"} Nov 03 20:32:53.800900 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.800734 4918 scope.go:122] "RemoveContainer" containerID="d6b8b9d717059d83f58b8196843dde345caf04f005df518a62bae98cb105637d" Nov 03 20:32:53.813104 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.813079 4918 scope.go:122] "RemoveContainer" containerID="d6b8b9d717059d83f58b8196843dde345caf04f005df518a62bae98cb105637d" Nov 03 20:32:53.813389 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:32:53.813371 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b8b9d717059d83f58b8196843dde345caf04f005df518a62bae98cb105637d\": container with ID starting with d6b8b9d717059d83f58b8196843dde345caf04f005df518a62bae98cb105637d not found: ID does not exist" containerID="d6b8b9d717059d83f58b8196843dde345caf04f005df518a62bae98cb105637d" Nov 03 20:32:53.813471 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.813397 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b8b9d717059d83f58b8196843dde345caf04f005df518a62bae98cb105637d"} err="failed to get container status \"d6b8b9d717059d83f58b8196843dde345caf04f005df518a62bae98cb105637d\": rpc error: code = NotFound desc = could not find container \"d6b8b9d717059d83f58b8196843dde345caf04f005df518a62bae98cb105637d\": container with ID starting with d6b8b9d717059d83f58b8196843dde345caf04f005df518a62bae98cb105637d not found: ID does not exist" Nov 03 20:32:53.827242 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.827212 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt"] Nov 03 20:32:53.832701 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:53.832679 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt"] Nov 03 20:32:54.396590 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:54.396535 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-5ca474/test-model-5ca474-kserve-5b7954d967-8hfgt" podUID="97238908-155f-4656-8767-fae2a876d653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.128:8000/ready\": context deadline exceeded" Nov 03 20:32:55.238746 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:55.238703 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97238908-155f-4656-8767-fae2a876d653" path="/var/lib/kubelet/pods/97238908-155f-4656-8767-fae2a876d653/volumes" Nov 03 20:32:55.924857 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:55.924821 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp"] Nov 03 20:32:55.936881 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:55.936844 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4"] Nov 03 20:32:56.843389 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:56.843317 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" podUID="b1203aec-cde6-46b4-acf9-991475721fd7" containerName="main" containerID="cri-o://a708094d8278074c3b0f07b6dac411f847d35d734a66472059ecfd770a0f5cc9" gracePeriod=45 Nov 03 20:32:56.952334 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:56.952293 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" podUID="b1203aec-cde6-46b4-acf9-991475721fd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": dial tcp 10.132.0.142:8000: connect: connection refused" Nov 03 20:32:57.392021 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.391993 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:32:57.439034 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.438997 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1203aec-cde6-46b4-acf9-991475721fd7-kube-api-access-2dcbw\" (UniqueName: \"kubernetes.io/projected/b1203aec-cde6-46b4-acf9-991475721fd7-kube-api-access-2dcbw\") pod \"b1203aec-cde6-46b4-acf9-991475721fd7\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " Nov 03 20:32:57.439290 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.439054 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-model-cache\") pod \"b1203aec-cde6-46b4-acf9-991475721fd7\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " Nov 03 20:32:57.439969 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.439267 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-tmp-dir\") pod \"b1203aec-cde6-46b4-acf9-991475721fd7\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " Nov 03 20:32:57.440134 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.439537 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-model-cache" pod "b1203aec-cde6-46b4-acf9-991475721fd7" (UID: "b1203aec-cde6-46b4-acf9-991475721fd7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:57.440134 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.439931 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-tmp-dir" pod "b1203aec-cde6-46b4-acf9-991475721fd7" (UID: "b1203aec-cde6-46b4-acf9-991475721fd7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:57.440360 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.440203 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-home\" (UniqueName: \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-home\") pod \"b1203aec-cde6-46b4-acf9-991475721fd7\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " Nov 03 20:32:57.440471 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.440436 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-home" pod "b1203aec-cde6-46b4-acf9-991475721fd7" (UID: "b1203aec-cde6-46b4-acf9-991475721fd7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:57.440711 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.440689 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1203aec-cde6-46b4-acf9-991475721fd7-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1203aec-cde6-46b4-acf9-991475721fd7-tls-certs\") pod \"b1203aec-cde6-46b4-acf9-991475721fd7\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " Nov 03 20:32:57.440711 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.440724 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-dshm\") pod \"b1203aec-cde6-46b4-acf9-991475721fd7\" (UID: \"b1203aec-cde6-46b4-acf9-991475721fd7\") " Nov 03 20:32:57.441606 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.441584 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-model-cache\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:57.441758 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.441735 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-tmp-dir\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:57.441881 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.441764 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-home\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:57.448835 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.448770 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-dshm" pod "b1203aec-cde6-46b4-acf9-991475721fd7" (UID: "b1203aec-cde6-46b4-acf9-991475721fd7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:57.448989 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.448965 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1203aec-cde6-46b4-acf9-991475721fd7-kube-api-access-2dcbw" pod "b1203aec-cde6-46b4-acf9-991475721fd7" (UID: "b1203aec-cde6-46b4-acf9-991475721fd7"). InnerVolumeSpecName "kube-api-access-2dcbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:57.449471 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.449412 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1203aec-cde6-46b4-acf9-991475721fd7-tls-certs" pod "b1203aec-cde6-46b4-acf9-991475721fd7" (UID: "b1203aec-cde6-46b4-acf9-991475721fd7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:57.543351 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.543267 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dcbw\" (UniqueName: \"kubernetes.io/projected/b1203aec-cde6-46b4-acf9-991475721fd7-kube-api-access-2dcbw\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:57.543351 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.543307 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1203aec-cde6-46b4-acf9-991475721fd7-tls-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:57.543351 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.543322 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1203aec-cde6-46b4-acf9-991475721fd7-dshm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:57.833411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.833377 4918 generic.go:356] "Generic (PLEG): container finished" podID="b1203aec-cde6-46b4-acf9-991475721fd7" containerID="a708094d8278074c3b0f07b6dac411f847d35d734a66472059ecfd770a0f5cc9" exitCode=0 Nov 03 20:32:57.833593 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.833423 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" Nov 03 20:32:57.833593 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.833463 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" event={"ID":"b1203aec-cde6-46b4-acf9-991475721fd7","Type":"ContainerDied","Data":"a708094d8278074c3b0f07b6dac411f847d35d734a66472059ecfd770a0f5cc9"} Nov 03 20:32:57.833593 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.833507 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92" event={"ID":"b1203aec-cde6-46b4-acf9-991475721fd7","Type":"ContainerDied","Data":"7ae43ead503a93a31cddd389119c61bb476b2bb6024a5d8d38ac5ce06e860be4"} Nov 03 20:32:57.833593 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.833524 4918 scope.go:122] "RemoveContainer" containerID="a708094d8278074c3b0f07b6dac411f847d35d734a66472059ecfd770a0f5cc9" Nov 03 20:32:57.845679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.845508 4918 scope.go:122] "RemoveContainer" containerID="a708094d8278074c3b0f07b6dac411f847d35d734a66472059ecfd770a0f5cc9" Nov 03 20:32:57.845751 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:32:57.845728 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a708094d8278074c3b0f07b6dac411f847d35d734a66472059ecfd770a0f5cc9\": container with ID starting with a708094d8278074c3b0f07b6dac411f847d35d734a66472059ecfd770a0f5cc9 not found: ID does not exist" containerID="a708094d8278074c3b0f07b6dac411f847d35d734a66472059ecfd770a0f5cc9" Nov 03 20:32:57.845806 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.845756 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a708094d8278074c3b0f07b6dac411f847d35d734a66472059ecfd770a0f5cc9"} err="failed to get container status \"a708094d8278074c3b0f07b6dac411f847d35d734a66472059ecfd770a0f5cc9\": rpc error: code = NotFound desc = could not find container \"a708094d8278074c3b0f07b6dac411f847d35d734a66472059ecfd770a0f5cc9\": container with ID starting with a708094d8278074c3b0f07b6dac411f847d35d734a66472059ecfd770a0f5cc9 not found: ID does not exist" Nov 03 20:32:57.860519 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.860490 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92"] Nov 03 20:32:57.864368 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:57.864343 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d0898b/test-model-d0898b-kserve-7977b76d69-wtr92"] Nov 03 20:32:58.213246 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:32:58.213146 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:32:59.217538 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:32:59.217502 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1203aec-cde6-46b4-acf9-991475721fd7" path="/var/lib/kubelet/pods/b1203aec-cde6-46b4-acf9-991475721fd7/volumes" Nov 03 20:33:00.925940 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:00.925817 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp" podUID="7103c18e-609f-41ed-862a-13a571202b51" containerName="payload-processing" containerID="cri-o://f527d1f78e11b17d0d71188cd8c106993be24d7cc59f06a444605b031ef0babd" gracePeriod=25 Nov 03 20:33:00.948692 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:00.944248 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4" podUID="3ded5532-c475-49a9-8e49-d4f547837159" containerName="payload-pre-processing" containerID="cri-o://c8d2d14f271483b1896b4c159a76ca419269f25f8e36e756a3c452d975dbfcdd" gracePeriod=25 Nov 03 20:33:06.633390 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:06.633349 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d"] Nov 03 20:33:06.633923 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:06.633620 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" podUID="e925174a-652a-4443-bf13-7b979933a297" containerName="istio-proxy" containerID="cri-o://d276de9b945efeb594e600ea2653f8663b24eb1361b8d6a361630e88cdb3f249" gracePeriod=30 Nov 03 20:33:10.687057 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:10.686965 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp" podUID="7103c18e-609f-41ed-862a-13a571202b51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:10.687726 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:10.687321 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4" podUID="3ded5532-c475-49a9-8e49-d4f547837159" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:11.971650 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:11.971576 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:33:11.994472 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:11.994442 4918 generic.go:356] "Generic (PLEG): container finished" podID="e925174a-652a-4443-bf13-7b979933a297" containerID="d276de9b945efeb594e600ea2653f8663b24eb1361b8d6a361630e88cdb3f249" exitCode=0 Nov 03 20:33:11.994641 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:11.994505 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" Nov 03 20:33:11.994641 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:11.994531 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" event={"ID":"e925174a-652a-4443-bf13-7b979933a297","Type":"ContainerDied","Data":"d276de9b945efeb594e600ea2653f8663b24eb1361b8d6a361630e88cdb3f249"} Nov 03 20:33:11.994641 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:11.994583 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d" event={"ID":"e925174a-652a-4443-bf13-7b979933a297","Type":"ContainerDied","Data":"d5441f7469dcd0ee46fc28322382a75479f6898e704fafe74d5d3a4713981fcb"} Nov 03 20:33:11.994641 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:11.994638 4918 scope.go:122] "RemoveContainer" containerID="d276de9b945efeb594e600ea2653f8663b24eb1361b8d6a361630e88cdb3f249" Nov 03 20:33:12.013256 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.013229 4918 scope.go:122] "RemoveContainer" containerID="d276de9b945efeb594e600ea2653f8663b24eb1361b8d6a361630e88cdb3f249" Nov 03 20:33:12.013632 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:33:12.013607 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d276de9b945efeb594e600ea2653f8663b24eb1361b8d6a361630e88cdb3f249\": container with ID starting with d276de9b945efeb594e600ea2653f8663b24eb1361b8d6a361630e88cdb3f249 not found: ID does not exist" containerID="d276de9b945efeb594e600ea2653f8663b24eb1361b8d6a361630e88cdb3f249" Nov 03 20:33:12.013728 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.013641 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d276de9b945efeb594e600ea2653f8663b24eb1361b8d6a361630e88cdb3f249"} err="failed to get container status \"d276de9b945efeb594e600ea2653f8663b24eb1361b8d6a361630e88cdb3f249\": rpc error: code = NotFound desc = could not find container \"d276de9b945efeb594e600ea2653f8663b24eb1361b8d6a361630e88cdb3f249\": container with ID starting with d276de9b945efeb594e600ea2653f8663b24eb1361b8d6a361630e88cdb3f249 not found: ID does not exist" Nov 03 20:33:12.066282 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.066237 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e925174a-652a-4443-bf13-7b979933a297-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e925174a-652a-4443-bf13-7b979933a297-istiod-ca-cert\") pod \"e925174a-652a-4443-bf13-7b979933a297\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " Nov 03 20:33:12.066479 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.066318 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e925174a-652a-4443-bf13-7b979933a297-istio-token\" (UniqueName: \"kubernetes.io/projected/e925174a-652a-4443-bf13-7b979933a297-istio-token\") pod \"e925174a-652a-4443-bf13-7b979933a297\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " Nov 03 20:33:12.066479 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.066361 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e925174a-652a-4443-bf13-7b979933a297-kube-api-access-cvrtd\" (UniqueName: \"kubernetes.io/projected/e925174a-652a-4443-bf13-7b979933a297-kube-api-access-cvrtd\") pod \"e925174a-652a-4443-bf13-7b979933a297\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " Nov 03 20:33:12.066479 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.066427 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-workload-socket\") pod \"e925174a-652a-4443-bf13-7b979933a297\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " Nov 03 20:33:12.066650 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.066509 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-credential-socket\") pod \"e925174a-652a-4443-bf13-7b979933a297\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " Nov 03 20:33:12.066650 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.066585 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-istio-envoy\") pod \"e925174a-652a-4443-bf13-7b979933a297\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " Nov 03 20:33:12.066650 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.066625 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e925174a-652a-4443-bf13-7b979933a297-istiod-ca-cert" pod "e925174a-652a-4443-bf13-7b979933a297" (UID: "e925174a-652a-4443-bf13-7b979933a297"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:12.066843 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.066793 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-workload-socket" pod "e925174a-652a-4443-bf13-7b979933a297" (UID: "e925174a-652a-4443-bf13-7b979933a297"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.066901 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.066875 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-credential-socket" pod "e925174a-652a-4443-bf13-7b979933a297" (UID: "e925174a-652a-4443-bf13-7b979933a297"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.067072 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.067049 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-istio-data\") pod \"e925174a-652a-4443-bf13-7b979933a297\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " Nov 03 20:33:12.067173 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.067128 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-workload-certs\") pod \"e925174a-652a-4443-bf13-7b979933a297\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " Nov 03 20:33:12.067250 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.067211 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e925174a-652a-4443-bf13-7b979933a297-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e925174a-652a-4443-bf13-7b979933a297-istio-podinfo\") pod \"e925174a-652a-4443-bf13-7b979933a297\" (UID: \"e925174a-652a-4443-bf13-7b979933a297\") " Nov 03 20:33:12.067425 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.067396 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-istio-data" pod "e925174a-652a-4443-bf13-7b979933a297" (UID: "e925174a-652a-4443-bf13-7b979933a297"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.067523 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.067420 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-workload-certs" pod "e925174a-652a-4443-bf13-7b979933a297" (UID: "e925174a-652a-4443-bf13-7b979933a297"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.068032 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.068008 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e925174a-652a-4443-bf13-7b979933a297-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.068137 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.068038 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.068137 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.068055 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.068137 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.068069 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.068137 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.068084 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.069236 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.069207 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e925174a-652a-4443-bf13-7b979933a297-istio-token" pod "e925174a-652a-4443-bf13-7b979933a297" (UID: "e925174a-652a-4443-bf13-7b979933a297"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:12.069347 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.069327 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e925174a-652a-4443-bf13-7b979933a297-kube-api-access-cvrtd" pod "e925174a-652a-4443-bf13-7b979933a297" (UID: "e925174a-652a-4443-bf13-7b979933a297"). InnerVolumeSpecName "kube-api-access-cvrtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:12.069530 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.069498 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-istio-envoy" pod "e925174a-652a-4443-bf13-7b979933a297" (UID: "e925174a-652a-4443-bf13-7b979933a297"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.069893 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.069869 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e925174a-652a-4443-bf13-7b979933a297-istio-podinfo" pod "e925174a-652a-4443-bf13-7b979933a297" (UID: "e925174a-652a-4443-bf13-7b979933a297"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:12.170379 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.170295 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e925174a-652a-4443-bf13-7b979933a297-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.170379 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.170325 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e925174a-652a-4443-bf13-7b979933a297-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.170379 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.170335 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e925174a-652a-4443-bf13-7b979933a297-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.170379 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.170345 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvrtd\" (UniqueName: \"kubernetes.io/projected/e925174a-652a-4443-bf13-7b979933a297-kube-api-access-cvrtd\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.337330 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.337276 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d"] Nov 03 20:33:12.342097 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:12.342059 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-wp95d"] Nov 03 20:33:13.216780 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:13.216743 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e925174a-652a-4443-bf13-7b979933a297" path="/var/lib/kubelet/pods/e925174a-652a-4443-bf13-7b979933a297/volumes" Nov 03 20:33:17.355470 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:17.355225 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8"] Nov 03 20:33:20.685441 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:20.685392 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4" podUID="3ded5532-c475-49a9-8e49-d4f547837159" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:20.686103 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:20.686064 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp" podUID="7103c18e-609f-41ed-862a-13a571202b51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:21.114914 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:21.114882 4918 generic.go:356] "Generic (PLEG): container finished" podID="3ded5532-c475-49a9-8e49-d4f547837159" containerID="c8d2d14f271483b1896b4c159a76ca419269f25f8e36e756a3c452d975dbfcdd" exitCode=0 Nov 03 20:33:21.115071 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:21.114959 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4" event={"ID":"3ded5532-c475-49a9-8e49-d4f547837159","Type":"ContainerDied","Data":"c8d2d14f271483b1896b4c159a76ca419269f25f8e36e756a3c452d975dbfcdd"} Nov 03 20:33:21.179234 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:21.178657 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4" Nov 03 20:33:21.228439 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:21.228361 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ded5532-c475-49a9-8e49-d4f547837159-kube-api-access-phl2r\" (UniqueName: \"kubernetes.io/projected/3ded5532-c475-49a9-8e49-d4f547837159-kube-api-access-phl2r\") pod \"3ded5532-c475-49a9-8e49-d4f547837159\" (UID: \"3ded5532-c475-49a9-8e49-d4f547837159\") " Nov 03 20:33:21.228680 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:21.228657 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ded5532-c475-49a9-8e49-d4f547837159-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ded5532-c475-49a9-8e49-d4f547837159-plugins-config-volume\") pod \"3ded5532-c475-49a9-8e49-d4f547837159\" (UID: \"3ded5532-c475-49a9-8e49-d4f547837159\") " Nov 03 20:33:21.229100 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:21.229068 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ded5532-c475-49a9-8e49-d4f547837159-plugins-config-volume" pod "3ded5532-c475-49a9-8e49-d4f547837159" (UID: "3ded5532-c475-49a9-8e49-d4f547837159"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:21.229409 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:21.229352 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ded5532-c475-49a9-8e49-d4f547837159-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:21.231227 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:21.231189 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ded5532-c475-49a9-8e49-d4f547837159-kube-api-access-phl2r" pod "3ded5532-c475-49a9-8e49-d4f547837159" (UID: "3ded5532-c475-49a9-8e49-d4f547837159"). InnerVolumeSpecName "kube-api-access-phl2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:21.288405 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:21.288371 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp" Nov 03 20:33:21.330426 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:21.330388 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7103c18e-609f-41ed-862a-13a571202b51-kube-api-access-gknq8\" (UniqueName: \"kubernetes.io/projected/7103c18e-609f-41ed-862a-13a571202b51-kube-api-access-gknq8\") pod \"7103c18e-609f-41ed-862a-13a571202b51\" (UID: \"7103c18e-609f-41ed-862a-13a571202b51\") " Nov 03 20:33:21.330672 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:21.330654 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7103c18e-609f-41ed-862a-13a571202b51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7103c18e-609f-41ed-862a-13a571202b51-plugins-config-volume\") pod \"7103c18e-609f-41ed-862a-13a571202b51\" (UID: \"7103c18e-609f-41ed-862a-13a571202b51\") " Nov 03 20:33:21.331140 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:21.331111 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7103c18e-609f-41ed-862a-13a571202b51-plugins-config-volume" pod "7103c18e-609f-41ed-862a-13a571202b51" (UID: "7103c18e-609f-41ed-862a-13a571202b51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:21.331521 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:21.331495 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7103c18e-609f-41ed-862a-13a571202b51-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:21.331594 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:21.331527 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phl2r\" (UniqueName: \"kubernetes.io/projected/3ded5532-c475-49a9-8e49-d4f547837159-kube-api-access-phl2r\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:21.332807 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:21.332768 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7103c18e-609f-41ed-862a-13a571202b51-kube-api-access-gknq8" pod "7103c18e-609f-41ed-862a-13a571202b51" (UID: "7103c18e-609f-41ed-862a-13a571202b51"). InnerVolumeSpecName "kube-api-access-gknq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:21.433313 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:21.433284 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gknq8\" (UniqueName: \"kubernetes.io/projected/7103c18e-609f-41ed-862a-13a571202b51-kube-api-access-gknq8\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:22.123737 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:22.123704 4918 generic.go:356] "Generic (PLEG): container finished" podID="7103c18e-609f-41ed-862a-13a571202b51" containerID="f527d1f78e11b17d0d71188cd8c106993be24d7cc59f06a444605b031ef0babd" exitCode=0 Nov 03 20:33:22.124224 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:22.123788 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp" event={"ID":"7103c18e-609f-41ed-862a-13a571202b51","Type":"ContainerDied","Data":"f527d1f78e11b17d0d71188cd8c106993be24d7cc59f06a444605b031ef0babd"} Nov 03 20:33:22.124224 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:22.123816 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp" Nov 03 20:33:22.124224 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:22.123851 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp" event={"ID":"7103c18e-609f-41ed-862a-13a571202b51","Type":"ContainerDied","Data":"ad4839d0c4654601252b935fd886441067c3af98fa9af1b00714350ef6a2a17d"} Nov 03 20:33:22.124224 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:22.123884 4918 scope.go:122] "RemoveContainer" containerID="f527d1f78e11b17d0d71188cd8c106993be24d7cc59f06a444605b031ef0babd" Nov 03 20:33:22.125453 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:22.125428 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4" Nov 03 20:33:22.125453 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:22.125444 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4" event={"ID":"3ded5532-c475-49a9-8e49-d4f547837159","Type":"ContainerDied","Data":"27181ea6b0fb1b8c946c232dfa53c9349d5b2f4b79b157df6e2289a326e09e65"} Nov 03 20:33:22.137008 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:22.136990 4918 scope.go:122] "RemoveContainer" containerID="f527d1f78e11b17d0d71188cd8c106993be24d7cc59f06a444605b031ef0babd" Nov 03 20:33:22.137308 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:33:22.137270 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f527d1f78e11b17d0d71188cd8c106993be24d7cc59f06a444605b031ef0babd\": container with ID starting with f527d1f78e11b17d0d71188cd8c106993be24d7cc59f06a444605b031ef0babd not found: ID does not exist" containerID="f527d1f78e11b17d0d71188cd8c106993be24d7cc59f06a444605b031ef0babd" Nov 03 20:33:22.137386 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:22.137308 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f527d1f78e11b17d0d71188cd8c106993be24d7cc59f06a444605b031ef0babd"} err="failed to get container status \"f527d1f78e11b17d0d71188cd8c106993be24d7cc59f06a444605b031ef0babd\": rpc error: code = NotFound desc = could not find container \"f527d1f78e11b17d0d71188cd8c106993be24d7cc59f06a444605b031ef0babd\": container with ID starting with f527d1f78e11b17d0d71188cd8c106993be24d7cc59f06a444605b031ef0babd not found: ID does not exist" Nov 03 20:33:22.137386 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:22.137334 4918 scope.go:122] "RemoveContainer" containerID="c8d2d14f271483b1896b4c159a76ca419269f25f8e36e756a3c452d975dbfcdd" Nov 03 20:33:22.157733 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:22.157712 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4"] Nov 03 20:33:22.161460 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:22.161439 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8c684d-9c9fcbdc9-g4dp4"] Nov 03 20:33:22.168923 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:22.168901 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp"] Nov 03 20:33:22.172792 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:22.172771 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c684d-6f5dc55ccd-q62lp"] Nov 03 20:33:22.355852 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:22.355813 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8" podUID="3b00acc2-340d-4fec-bab6-efadbf64e74d" containerName="payload-processing" containerID="cri-o://bc79b38baca616aaf2ed70277f71ab50d22358422abd1271a95cfef8f52e6551" gracePeriod=25 Nov 03 20:33:22.675806 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:22.675703 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz"] Nov 03 20:33:22.675975 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:22.675958 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" podUID="eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" containerName="istio-proxy" containerID="cri-o://65bfdb998892e61eafa3dcc00a87bffc1735424b8d82976a086b8e4bd5b860aa" gracePeriod=30 Nov 03 20:33:23.216590 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:23.216555 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ded5532-c475-49a9-8e49-d4f547837159" path="/var/lib/kubelet/pods/3ded5532-c475-49a9-8e49-d4f547837159/volumes" Nov 03 20:33:23.217304 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:23.217283 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7103c18e-609f-41ed-862a-13a571202b51" path="/var/lib/kubelet/pods/7103c18e-609f-41ed-862a-13a571202b51/volumes" Nov 03 20:33:27.967468 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:27.967406 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:33:28.059890 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.059853 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-token\" (UniqueName: \"kubernetes.io/projected/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-token\") pod \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " Nov 03 20:33:28.060095 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.060016 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-workload-socket\") pod \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " Nov 03 20:33:28.060197 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.060109 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-podinfo\") pod \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " Nov 03 20:33:28.060269 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.060250 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-data\") pod \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " Nov 03 20:33:28.060338 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.060313 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-workload-certs\") pod \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " Nov 03 20:33:28.060393 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.060345 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istiod-ca-cert\") pod \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " Nov 03 20:33:28.060447 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.060405 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-credential-socket\") pod \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " Nov 03 20:33:28.060510 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.060468 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-kube-api-access-f6mmj\" (UniqueName: \"kubernetes.io/projected/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-kube-api-access-f6mmj\") pod \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " Nov 03 20:33:28.060510 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.060502 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-envoy\") pod \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\" (UID: \"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5\") " Nov 03 20:33:28.061596 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.061560 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-workload-socket" pod "eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" (UID: "eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:28.061860 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.061824 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-credential-socket" pod "eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" (UID: "eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:28.061860 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.061837 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-workload-certs" pod "eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" (UID: "eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:28.063022 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.062721 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istiod-ca-cert" pod "eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" (UID: "eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:28.063462 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.063433 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-data" pod "eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" (UID: "eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:28.064513 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.064487 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-envoy" pod "eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" (UID: "eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:28.065893 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.065862 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-kube-api-access-f6mmj" pod "eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" (UID: "eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5"). InnerVolumeSpecName "kube-api-access-f6mmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:28.067198 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.067159 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-podinfo" pod "eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" (UID: "eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:28.067694 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.067673 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-token" pod "eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" (UID: "eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:28.165119 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.165083 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.165119 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.165118 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.165342 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.165130 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.165342 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.165139 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.165342 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.165151 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.165342 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.165160 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6mmj\" (UniqueName: \"kubernetes.io/projected/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-kube-api-access-f6mmj\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.165342 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.165170 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.165342 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.165195 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.165342 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.165205 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.175620 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.175588 4918 generic.go:356] "Generic (PLEG): container finished" podID="eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" containerID="65bfdb998892e61eafa3dcc00a87bffc1735424b8d82976a086b8e4bd5b860aa" exitCode=0 Nov 03 20:33:28.175762 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.175640 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" Nov 03 20:33:28.175762 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.175673 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" event={"ID":"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5","Type":"ContainerDied","Data":"65bfdb998892e61eafa3dcc00a87bffc1735424b8d82976a086b8e4bd5b860aa"} Nov 03 20:33:28.175762 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.175742 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz" event={"ID":"eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5","Type":"ContainerDied","Data":"735fb35d9dd9334ec06ca4579feaa69efc834eb531fa509413651cffaec7a222"} Nov 03 20:33:28.175917 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.175766 4918 scope.go:122] "RemoveContainer" containerID="65bfdb998892e61eafa3dcc00a87bffc1735424b8d82976a086b8e4bd5b860aa" Nov 03 20:33:28.187912 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.187890 4918 scope.go:122] "RemoveContainer" containerID="65bfdb998892e61eafa3dcc00a87bffc1735424b8d82976a086b8e4bd5b860aa" Nov 03 20:33:28.188168 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:33:28.188146 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65bfdb998892e61eafa3dcc00a87bffc1735424b8d82976a086b8e4bd5b860aa\": container with ID starting with 65bfdb998892e61eafa3dcc00a87bffc1735424b8d82976a086b8e4bd5b860aa not found: ID does not exist" containerID="65bfdb998892e61eafa3dcc00a87bffc1735424b8d82976a086b8e4bd5b860aa" Nov 03 20:33:28.188263 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.188190 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65bfdb998892e61eafa3dcc00a87bffc1735424b8d82976a086b8e4bd5b860aa"} err="failed to get container status \"65bfdb998892e61eafa3dcc00a87bffc1735424b8d82976a086b8e4bd5b860aa\": rpc error: code = NotFound desc = could not find container \"65bfdb998892e61eafa3dcc00a87bffc1735424b8d82976a086b8e4bd5b860aa\": container with ID starting with 65bfdb998892e61eafa3dcc00a87bffc1735424b8d82976a086b8e4bd5b860aa not found: ID does not exist" Nov 03 20:33:28.207906 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.207864 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz"] Nov 03 20:33:28.213230 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:33:28.213168 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:33:28.213868 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:28.213845 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-5zrjz"] Nov 03 20:33:29.221330 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:29.221294 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" path="/var/lib/kubelet/pods/eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5/volumes" Nov 03 20:33:29.221820 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:29.221777 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj"] Nov 03 20:33:29.222036 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:29.222012 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" podUID="769e99e9-f468-4747-ac31-5fe11e893dce" containerName="istio-proxy" containerID="cri-o://9c9821b5dab68e3bc6c65284882421a04f5832885bde022f8381bd42f82f6291" gracePeriod=30 Nov 03 20:33:30.834831 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:30.834785 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8" podUID="3b00acc2-340d-4fec-bab6-efadbf64e74d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:33.335710 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:33.335676 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc"] Nov 03 20:33:33.538588 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:33.538552 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b"] Nov 03 20:33:33.538827 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:33.538793 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" podUID="4b47d59c-60aa-483d-82bd-35a5af6b10b6" containerName="maas-api" containerID="cri-o://ad8a6f1fde75b55f94376445384ed67677a96db22d92248c86a4d8644364919e" gracePeriod=30 Nov 03 20:33:34.052509 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.052477 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" Nov 03 20:33:34.241215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.240922 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b47d59c-60aa-483d-82bd-35a5af6b10b6-kube-api-access-9v58x\" (UniqueName: \"kubernetes.io/projected/4b47d59c-60aa-483d-82bd-35a5af6b10b6-kube-api-access-9v58x\") pod \"4b47d59c-60aa-483d-82bd-35a5af6b10b6\" (UID: \"4b47d59c-60aa-483d-82bd-35a5af6b10b6\") " Nov 03 20:33:34.241215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.241009 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b47d59c-60aa-483d-82bd-35a5af6b10b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b47d59c-60aa-483d-82bd-35a5af6b10b6-metrics-certs\") pod \"4b47d59c-60aa-483d-82bd-35a5af6b10b6\" (UID: \"4b47d59c-60aa-483d-82bd-35a5af6b10b6\") " Nov 03 20:33:34.241215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.241088 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b47d59c-60aa-483d-82bd-35a5af6b10b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b47d59c-60aa-483d-82bd-35a5af6b10b6-openshift-service-ca\") pod \"4b47d59c-60aa-483d-82bd-35a5af6b10b6\" (UID: \"4b47d59c-60aa-483d-82bd-35a5af6b10b6\") " Nov 03 20:33:34.241215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.241163 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b47d59c-60aa-483d-82bd-35a5af6b10b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b47d59c-60aa-483d-82bd-35a5af6b10b6-maas-api-tls\") pod \"4b47d59c-60aa-483d-82bd-35a5af6b10b6\" (UID: \"4b47d59c-60aa-483d-82bd-35a5af6b10b6\") " Nov 03 20:33:34.269395 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.269308 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b47d59c-60aa-483d-82bd-35a5af6b10b6-openshift-service-ca" pod "4b47d59c-60aa-483d-82bd-35a5af6b10b6" (UID: "4b47d59c-60aa-483d-82bd-35a5af6b10b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:34.269595 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.269519 4918 generic.go:356] "Generic (PLEG): container finished" podID="4b47d59c-60aa-483d-82bd-35a5af6b10b6" containerID="ad8a6f1fde75b55f94376445384ed67677a96db22d92248c86a4d8644364919e" exitCode=0 Nov 03 20:33:34.269803 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.269546 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" event={"ID":"4b47d59c-60aa-483d-82bd-35a5af6b10b6","Type":"ContainerDied","Data":"ad8a6f1fde75b55f94376445384ed67677a96db22d92248c86a4d8644364919e"} Nov 03 20:33:34.285453 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.285406 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" event={"ID":"4b47d59c-60aa-483d-82bd-35a5af6b10b6","Type":"ContainerDied","Data":"89c2eaf05d465cc55b355382086a8bb6b74531ffacfa5fbfa3e0d294b0fd6c3c"} Nov 03 20:33:34.285785 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.270632 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b47d59c-60aa-483d-82bd-35a5af6b10b6-kube-api-access-9v58x" pod "4b47d59c-60aa-483d-82bd-35a5af6b10b6" (UID: "4b47d59c-60aa-483d-82bd-35a5af6b10b6"). InnerVolumeSpecName "kube-api-access-9v58x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:34.285999 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.269617 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b" Nov 03 20:33:34.286588 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.285461 4918 scope.go:122] "RemoveContainer" containerID="ad8a6f1fde75b55f94376445384ed67677a96db22d92248c86a4d8644364919e" Nov 03 20:33:34.301707 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.301664 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b47d59c-60aa-483d-82bd-35a5af6b10b6-metrics-certs" pod "4b47d59c-60aa-483d-82bd-35a5af6b10b6" (UID: "4b47d59c-60aa-483d-82bd-35a5af6b10b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:34.304222 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.304140 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b47d59c-60aa-483d-82bd-35a5af6b10b6-maas-api-tls" pod "4b47d59c-60aa-483d-82bd-35a5af6b10b6" (UID: "4b47d59c-60aa-483d-82bd-35a5af6b10b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:34.344095 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.344068 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v58x\" (UniqueName: \"kubernetes.io/projected/4b47d59c-60aa-483d-82bd-35a5af6b10b6-kube-api-access-9v58x\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:34.344095 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.344095 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b47d59c-60aa-483d-82bd-35a5af6b10b6-metrics-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:34.344759 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.344105 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b47d59c-60aa-483d-82bd-35a5af6b10b6-openshift-service-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:34.344759 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.344115 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b47d59c-60aa-483d-82bd-35a5af6b10b6-maas-api-tls\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:34.357719 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.357685 4918 scope.go:122] "RemoveContainer" containerID="ad8a6f1fde75b55f94376445384ed67677a96db22d92248c86a4d8644364919e" Nov 03 20:33:34.358160 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:33:34.358133 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad8a6f1fde75b55f94376445384ed67677a96db22d92248c86a4d8644364919e\": container with ID starting with ad8a6f1fde75b55f94376445384ed67677a96db22d92248c86a4d8644364919e not found: ID does not exist" containerID="ad8a6f1fde75b55f94376445384ed67677a96db22d92248c86a4d8644364919e" Nov 03 20:33:34.358699 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.358172 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad8a6f1fde75b55f94376445384ed67677a96db22d92248c86a4d8644364919e"} err="failed to get container status \"ad8a6f1fde75b55f94376445384ed67677a96db22d92248c86a4d8644364919e\": rpc error: code = NotFound desc = could not find container \"ad8a6f1fde75b55f94376445384ed67677a96db22d92248c86a4d8644364919e\": container with ID starting with ad8a6f1fde75b55f94376445384ed67677a96db22d92248c86a4d8644364919e not found: ID does not exist" Nov 03 20:33:34.510740 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.510333 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:33:34.617275 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.617243 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b"] Nov 03 20:33:34.626219 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.626171 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-90ef9c-5498778889-h885b"] Nov 03 20:33:34.651145 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.651111 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-istio-envoy\") pod \"769e99e9-f468-4747-ac31-5fe11e893dce\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " Nov 03 20:33:34.651307 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.651228 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769e99e9-f468-4747-ac31-5fe11e893dce-kube-api-access-h9s2q\" (UniqueName: \"kubernetes.io/projected/769e99e9-f468-4747-ac31-5fe11e893dce-kube-api-access-h9s2q\") pod \"769e99e9-f468-4747-ac31-5fe11e893dce\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " Nov 03 20:33:34.651307 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.651281 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/769e99e9-f468-4747-ac31-5fe11e893dce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769e99e9-f468-4747-ac31-5fe11e893dce-istiod-ca-cert\") pod \"769e99e9-f468-4747-ac31-5fe11e893dce\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " Nov 03 20:33:34.651465 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.651445 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-istio-data\") pod \"769e99e9-f468-4747-ac31-5fe11e893dce\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " Nov 03 20:33:34.651525 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.651495 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/769e99e9-f468-4747-ac31-5fe11e893dce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769e99e9-f468-4747-ac31-5fe11e893dce-istio-podinfo\") pod \"769e99e9-f468-4747-ac31-5fe11e893dce\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " Nov 03 20:33:34.651588 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.651542 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769e99e9-f468-4747-ac31-5fe11e893dce-istio-token\" (UniqueName: \"kubernetes.io/projected/769e99e9-f468-4747-ac31-5fe11e893dce-istio-token\") pod \"769e99e9-f468-4747-ac31-5fe11e893dce\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " Nov 03 20:33:34.651588 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.651566 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-credential-socket\") pod \"769e99e9-f468-4747-ac31-5fe11e893dce\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " Nov 03 20:33:34.651695 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.651603 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-workload-certs\") pod \"769e99e9-f468-4747-ac31-5fe11e893dce\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " Nov 03 20:33:34.651695 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.651670 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-workload-socket\") pod \"769e99e9-f468-4747-ac31-5fe11e893dce\" (UID: \"769e99e9-f468-4747-ac31-5fe11e893dce\") " Nov 03 20:33:34.652234 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.652207 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/769e99e9-f468-4747-ac31-5fe11e893dce-istiod-ca-cert" pod "769e99e9-f468-4747-ac31-5fe11e893dce" (UID: "769e99e9-f468-4747-ac31-5fe11e893dce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:34.652909 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.652447 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-workload-socket" pod "769e99e9-f468-4747-ac31-5fe11e893dce" (UID: "769e99e9-f468-4747-ac31-5fe11e893dce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:34.652909 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.652596 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-istio-data" pod "769e99e9-f468-4747-ac31-5fe11e893dce" (UID: "769e99e9-f468-4747-ac31-5fe11e893dce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:34.652909 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.652606 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769e99e9-f468-4747-ac31-5fe11e893dce-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:34.652909 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.652657 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:34.652909 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.652666 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-workload-certs" pod "769e99e9-f468-4747-ac31-5fe11e893dce" (UID: "769e99e9-f468-4747-ac31-5fe11e893dce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:34.653386 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.653272 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-credential-socket" pod "769e99e9-f468-4747-ac31-5fe11e893dce" (UID: "769e99e9-f468-4747-ac31-5fe11e893dce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:34.653887 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.653853 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-istio-envoy" pod "769e99e9-f468-4747-ac31-5fe11e893dce" (UID: "769e99e9-f468-4747-ac31-5fe11e893dce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:34.654008 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.653975 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769e99e9-f468-4747-ac31-5fe11e893dce-kube-api-access-h9s2q" pod "769e99e9-f468-4747-ac31-5fe11e893dce" (UID: "769e99e9-f468-4747-ac31-5fe11e893dce"). InnerVolumeSpecName "kube-api-access-h9s2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:34.654601 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.654568 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/769e99e9-f468-4747-ac31-5fe11e893dce-istio-podinfo" pod "769e99e9-f468-4747-ac31-5fe11e893dce" (UID: "769e99e9-f468-4747-ac31-5fe11e893dce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:34.655108 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.655083 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769e99e9-f468-4747-ac31-5fe11e893dce-istio-token" pod "769e99e9-f468-4747-ac31-5fe11e893dce" (UID: "769e99e9-f468-4747-ac31-5fe11e893dce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:34.754106 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.754072 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:34.754106 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.754106 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:34.754353 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.754117 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9s2q\" (UniqueName: \"kubernetes.io/projected/769e99e9-f468-4747-ac31-5fe11e893dce-kube-api-access-h9s2q\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:34.754353 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.754126 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:34.754353 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.754135 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769e99e9-f468-4747-ac31-5fe11e893dce-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:34.754353 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.754144 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769e99e9-f468-4747-ac31-5fe11e893dce-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:34.754353 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:34.754153 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e99e9-f468-4747-ac31-5fe11e893dce-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:35.218024 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:35.217985 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b47d59c-60aa-483d-82bd-35a5af6b10b6" path="/var/lib/kubelet/pods/4b47d59c-60aa-483d-82bd-35a5af6b10b6/volumes" Nov 03 20:33:35.278076 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:33:35.278039 4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769e99e9_f468_4747_ac31_5fe11e893dce.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769e99e9_f468_4747_ac31_5fe11e893dce.slice/crio-1a1d5c7ad97b35e224a670c35eec88e8af04f1f39e87cc27d781f61faff2e8ce\": RecentStats: unable to find data in memory cache]" Nov 03 20:33:35.281088 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:35.281059 4918 generic.go:356] "Generic (PLEG): container finished" podID="769e99e9-f468-4747-ac31-5fe11e893dce" containerID="9c9821b5dab68e3bc6c65284882421a04f5832885bde022f8381bd42f82f6291" exitCode=0 Nov 03 20:33:35.281250 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:35.281140 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" Nov 03 20:33:35.281250 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:35.281148 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" event={"ID":"769e99e9-f468-4747-ac31-5fe11e893dce","Type":"ContainerDied","Data":"9c9821b5dab68e3bc6c65284882421a04f5832885bde022f8381bd42f82f6291"} Nov 03 20:33:35.281392 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:35.281260 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj" event={"ID":"769e99e9-f468-4747-ac31-5fe11e893dce","Type":"ContainerDied","Data":"1a1d5c7ad97b35e224a670c35eec88e8af04f1f39e87cc27d781f61faff2e8ce"} Nov 03 20:33:35.281392 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:35.281293 4918 scope.go:122] "RemoveContainer" containerID="9c9821b5dab68e3bc6c65284882421a04f5832885bde022f8381bd42f82f6291" Nov 03 20:33:35.292929 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:35.292912 4918 scope.go:122] "RemoveContainer" containerID="9c9821b5dab68e3bc6c65284882421a04f5832885bde022f8381bd42f82f6291" Nov 03 20:33:35.293270 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:33:35.293250 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c9821b5dab68e3bc6c65284882421a04f5832885bde022f8381bd42f82f6291\": container with ID starting with 9c9821b5dab68e3bc6c65284882421a04f5832885bde022f8381bd42f82f6291 not found: ID does not exist" containerID="9c9821b5dab68e3bc6c65284882421a04f5832885bde022f8381bd42f82f6291" Nov 03 20:33:35.293341 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:35.293279 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c9821b5dab68e3bc6c65284882421a04f5832885bde022f8381bd42f82f6291"} err="failed to get container status \"9c9821b5dab68e3bc6c65284882421a04f5832885bde022f8381bd42f82f6291\": rpc error: code = NotFound desc = could not find container \"9c9821b5dab68e3bc6c65284882421a04f5832885bde022f8381bd42f82f6291\": container with ID starting with 9c9821b5dab68e3bc6c65284882421a04f5832885bde022f8381bd42f82f6291 not found: ID does not exist" Nov 03 20:33:35.303104 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:35.303081 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj"] Nov 03 20:33:35.309943 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:35.309920 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-sgvtj"] Nov 03 20:33:37.216360 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:37.216324 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="769e99e9-f468-4747-ac31-5fe11e893dce" path="/var/lib/kubelet/pods/769e99e9-f468-4747-ac31-5fe11e893dce/volumes" Nov 03 20:33:38.336419 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:38.336335 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc" podUID="b612e8de-dd1e-47db-a7f8-28eb9e4149f9" containerName="payload-processing" containerID="cri-o://c571d830d0a97278b8c9a56fd3e53d62cf83d796e7b90c1532b4e4af80e59904" gracePeriod=25 Nov 03 20:33:38.610927 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:38.610898 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc" Nov 03 20:33:38.714897 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:38.714863 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b612e8de-dd1e-47db-a7f8-28eb9e4149f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b612e8de-dd1e-47db-a7f8-28eb9e4149f9-plugins-config-volume\") pod \"b612e8de-dd1e-47db-a7f8-28eb9e4149f9\" (UID: \"b612e8de-dd1e-47db-a7f8-28eb9e4149f9\") " Nov 03 20:33:38.715103 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:38.715023 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b612e8de-dd1e-47db-a7f8-28eb9e4149f9-kube-api-access-rbmzs\" (UniqueName: \"kubernetes.io/projected/b612e8de-dd1e-47db-a7f8-28eb9e4149f9-kube-api-access-rbmzs\") pod \"b612e8de-dd1e-47db-a7f8-28eb9e4149f9\" (UID: \"b612e8de-dd1e-47db-a7f8-28eb9e4149f9\") " Nov 03 20:33:38.715369 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:38.715341 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b612e8de-dd1e-47db-a7f8-28eb9e4149f9-plugins-config-volume" pod "b612e8de-dd1e-47db-a7f8-28eb9e4149f9" (UID: "b612e8de-dd1e-47db-a7f8-28eb9e4149f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:38.715735 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:38.715715 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b612e8de-dd1e-47db-a7f8-28eb9e4149f9-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:38.718391 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:38.718356 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b612e8de-dd1e-47db-a7f8-28eb9e4149f9-kube-api-access-rbmzs" pod "b612e8de-dd1e-47db-a7f8-28eb9e4149f9" (UID: "b612e8de-dd1e-47db-a7f8-28eb9e4149f9"). InnerVolumeSpecName "kube-api-access-rbmzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:38.817210 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:38.817145 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbmzs\" (UniqueName: \"kubernetes.io/projected/b612e8de-dd1e-47db-a7f8-28eb9e4149f9-kube-api-access-rbmzs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:39.315874 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:39.315840 4918 generic.go:356] "Generic (PLEG): container finished" podID="b612e8de-dd1e-47db-a7f8-28eb9e4149f9" containerID="c571d830d0a97278b8c9a56fd3e53d62cf83d796e7b90c1532b4e4af80e59904" exitCode=0 Nov 03 20:33:39.316066 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:39.315902 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc" Nov 03 20:33:39.316066 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:39.315925 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc" event={"ID":"b612e8de-dd1e-47db-a7f8-28eb9e4149f9","Type":"ContainerDied","Data":"c571d830d0a97278b8c9a56fd3e53d62cf83d796e7b90c1532b4e4af80e59904"} Nov 03 20:33:39.316066 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:39.315990 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc" event={"ID":"b612e8de-dd1e-47db-a7f8-28eb9e4149f9","Type":"ContainerDied","Data":"1a5fcd1c0bca7f1186e92f52ebde7137d10a0d500e1f9a31bd91c4314618c5ae"} Nov 03 20:33:39.316066 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:39.316014 4918 scope.go:122] "RemoveContainer" containerID="c571d830d0a97278b8c9a56fd3e53d62cf83d796e7b90c1532b4e4af80e59904" Nov 03 20:33:39.328044 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:39.328025 4918 scope.go:122] "RemoveContainer" containerID="c571d830d0a97278b8c9a56fd3e53d62cf83d796e7b90c1532b4e4af80e59904" Nov 03 20:33:39.328332 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:33:39.328310 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c571d830d0a97278b8c9a56fd3e53d62cf83d796e7b90c1532b4e4af80e59904\": container with ID starting with c571d830d0a97278b8c9a56fd3e53d62cf83d796e7b90c1532b4e4af80e59904 not found: ID does not exist" containerID="c571d830d0a97278b8c9a56fd3e53d62cf83d796e7b90c1532b4e4af80e59904" Nov 03 20:33:39.328393 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:39.328341 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c571d830d0a97278b8c9a56fd3e53d62cf83d796e7b90c1532b4e4af80e59904"} err="failed to get container status \"c571d830d0a97278b8c9a56fd3e53d62cf83d796e7b90c1532b4e4af80e59904\": rpc error: code = NotFound desc = could not find container \"c571d830d0a97278b8c9a56fd3e53d62cf83d796e7b90c1532b4e4af80e59904\": container with ID starting with c571d830d0a97278b8c9a56fd3e53d62cf83d796e7b90c1532b4e4af80e59904 not found: ID does not exist" Nov 03 20:33:39.337784 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:39.337760 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc"] Nov 03 20:33:39.341280 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:39.341249 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b0adda-676477c9db-4nqkc"] Nov 03 20:33:39.836553 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:39.836515 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw"] Nov 03 20:33:40.464973 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:40.464931 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc"] Nov 03 20:33:40.465559 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:40.465301 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" podUID="b2461bee-954b-481d-a740-379482984a4a" containerName="istio-proxy" containerID="cri-o://d63bf761de529013fca363edfa148446ae91f5a1eab8e6c77b46173101569dcf" gracePeriod=30 Nov 03 20:33:40.472353 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:40.472322 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj"] Nov 03 20:33:40.472603 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:40.472570 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" podUID="b6290d9f-ddd4-40d2-809b-110249e01125" containerName="istio-proxy" containerID="cri-o://3f90c4195c6cc9050988f91e71616e07936195cfa707d38e8fda88008ad2f111" gracePeriod=30 Nov 03 20:33:40.836413 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:40.836373 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8" podUID="3b00acc2-340d-4fec-bab6-efadbf64e74d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:41.215974 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:41.215899 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b612e8de-dd1e-47db-a7f8-28eb9e4149f9" path="/var/lib/kubelet/pods/b612e8de-dd1e-47db-a7f8-28eb9e4149f9/volumes" Nov 03 20:33:42.307054 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:42.307013 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk"] Nov 03 20:33:42.307576 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:42.307262 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" podUID="ea246681-cf7a-40b9-97d1-9c1d2594c3ec" containerName="istio-proxy" containerID="cri-o://5ee2fb2cb5bab41990dacff40ca3ad55ce1d6623450c2264e1a870d6db532581" gracePeriod=30 Nov 03 20:33:42.739934 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:42.739897 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8" Nov 03 20:33:42.826091 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:42.826053 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b00acc2-340d-4fec-bab6-efadbf64e74d-kube-api-access-tgtlm\" (UniqueName: \"kubernetes.io/projected/3b00acc2-340d-4fec-bab6-efadbf64e74d-kube-api-access-tgtlm\") pod \"3b00acc2-340d-4fec-bab6-efadbf64e74d\" (UID: \"3b00acc2-340d-4fec-bab6-efadbf64e74d\") " Nov 03 20:33:42.826268 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:42.826111 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b00acc2-340d-4fec-bab6-efadbf64e74d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b00acc2-340d-4fec-bab6-efadbf64e74d-plugins-config-volume\") pod \"3b00acc2-340d-4fec-bab6-efadbf64e74d\" (UID: \"3b00acc2-340d-4fec-bab6-efadbf64e74d\") " Nov 03 20:33:42.826591 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:42.826561 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b00acc2-340d-4fec-bab6-efadbf64e74d-plugins-config-volume" pod "3b00acc2-340d-4fec-bab6-efadbf64e74d" (UID: "3b00acc2-340d-4fec-bab6-efadbf64e74d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:42.826941 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:42.826916 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b00acc2-340d-4fec-bab6-efadbf64e74d-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:42.828285 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:42.828258 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b00acc2-340d-4fec-bab6-efadbf64e74d-kube-api-access-tgtlm" pod "3b00acc2-340d-4fec-bab6-efadbf64e74d" (UID: "3b00acc2-340d-4fec-bab6-efadbf64e74d"). InnerVolumeSpecName "kube-api-access-tgtlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:42.928268 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:42.928234 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgtlm\" (UniqueName: \"kubernetes.io/projected/3b00acc2-340d-4fec-bab6-efadbf64e74d-kube-api-access-tgtlm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:43.082990 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:43.082954 4918 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:33:43.083168 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:43.083016 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" podUID="ea246681-cf7a-40b9-97d1-9c1d2594c3ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:33:43.374245 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:43.374151 4918 generic.go:356] "Generic (PLEG): container finished" podID="3b00acc2-340d-4fec-bab6-efadbf64e74d" containerID="bc79b38baca616aaf2ed70277f71ab50d22358422abd1271a95cfef8f52e6551" exitCode=0 Nov 03 20:33:43.374670 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:43.374241 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8" Nov 03 20:33:43.374670 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:43.374247 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8" event={"ID":"3b00acc2-340d-4fec-bab6-efadbf64e74d","Type":"ContainerDied","Data":"bc79b38baca616aaf2ed70277f71ab50d22358422abd1271a95cfef8f52e6551"} Nov 03 20:33:43.374670 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:43.374310 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8" event={"ID":"3b00acc2-340d-4fec-bab6-efadbf64e74d","Type":"ContainerDied","Data":"3f0132287cf9b4d20bd505f2d10f95b0aa7ca91681fc109a89c56cd71ae70401"} Nov 03 20:33:43.374670 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:43.374341 4918 scope.go:122] "RemoveContainer" containerID="bc79b38baca616aaf2ed70277f71ab50d22358422abd1271a95cfef8f52e6551" Nov 03 20:33:43.385988 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:43.385963 4918 scope.go:122] "RemoveContainer" containerID="bc79b38baca616aaf2ed70277f71ab50d22358422abd1271a95cfef8f52e6551" Nov 03 20:33:43.386326 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:33:43.386303 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc79b38baca616aaf2ed70277f71ab50d22358422abd1271a95cfef8f52e6551\": container with ID starting with bc79b38baca616aaf2ed70277f71ab50d22358422abd1271a95cfef8f52e6551 not found: ID does not exist" containerID="bc79b38baca616aaf2ed70277f71ab50d22358422abd1271a95cfef8f52e6551" Nov 03 20:33:43.386417 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:43.386332 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc79b38baca616aaf2ed70277f71ab50d22358422abd1271a95cfef8f52e6551"} err="failed to get container status \"bc79b38baca616aaf2ed70277f71ab50d22358422abd1271a95cfef8f52e6551\": rpc error: code = NotFound desc = could not find container \"bc79b38baca616aaf2ed70277f71ab50d22358422abd1271a95cfef8f52e6551\": container with ID starting with bc79b38baca616aaf2ed70277f71ab50d22358422abd1271a95cfef8f52e6551 not found: ID does not exist" Nov 03 20:33:43.397123 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:43.397094 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8"] Nov 03 20:33:43.400516 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:43.400487 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ca474-55b4b495c8-ghqj8"] Nov 03 20:33:43.670255 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:43.670131 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb"] Nov 03 20:33:43.670417 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:43.670392 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" podUID="14e66787-45ea-4e62-83f8-e14a0388b915" containerName="maas-api" containerID="cri-o://adc37e0e1df0f755ccac79a67244d11d4338893f1a82296b1eaf9b814506a480" gracePeriod=30 Nov 03 20:33:43.764046 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:33:43.764008 4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e66787_45ea_4e62_83f8_e14a0388b915.slice/crio-adc37e0e1df0f755ccac79a67244d11d4338893f1a82296b1eaf9b814506a480.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e66787_45ea_4e62_83f8_e14a0388b915.slice/crio-conmon-adc37e0e1df0f755ccac79a67244d11d4338893f1a82296b1eaf9b814506a480.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:33:44.095597 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.095545 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" Nov 03 20:33:44.216169 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.216138 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14e66787-45ea-4e62-83f8-e14a0388b915-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e66787-45ea-4e62-83f8-e14a0388b915-openshift-service-ca\") pod \"14e66787-45ea-4e62-83f8-e14a0388b915\" (UID: \"14e66787-45ea-4e62-83f8-e14a0388b915\") " Nov 03 20:33:44.216406 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.216233 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14e66787-45ea-4e62-83f8-e14a0388b915-metrics-certs\" (UniqueName: \"kubernetes.io/secret/14e66787-45ea-4e62-83f8-e14a0388b915-metrics-certs\") pod \"14e66787-45ea-4e62-83f8-e14a0388b915\" (UID: \"14e66787-45ea-4e62-83f8-e14a0388b915\") " Nov 03 20:33:44.216475 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.216441 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14e66787-45ea-4e62-83f8-e14a0388b915-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14e66787-45ea-4e62-83f8-e14a0388b915-maas-api-tls\") pod \"14e66787-45ea-4e62-83f8-e14a0388b915\" (UID: \"14e66787-45ea-4e62-83f8-e14a0388b915\") " Nov 03 20:33:44.216532 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.216487 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e66787-45ea-4e62-83f8-e14a0388b915-openshift-service-ca" pod "14e66787-45ea-4e62-83f8-e14a0388b915" (UID: "14e66787-45ea-4e62-83f8-e14a0388b915"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:44.216584 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.216534 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14e66787-45ea-4e62-83f8-e14a0388b915-kube-api-access-tt7dd\" (UniqueName: \"kubernetes.io/projected/14e66787-45ea-4e62-83f8-e14a0388b915-kube-api-access-tt7dd\") pod \"14e66787-45ea-4e62-83f8-e14a0388b915\" (UID: \"14e66787-45ea-4e62-83f8-e14a0388b915\") " Nov 03 20:33:44.216955 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.216931 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e66787-45ea-4e62-83f8-e14a0388b915-openshift-service-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:44.218500 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.218470 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14e66787-45ea-4e62-83f8-e14a0388b915-metrics-certs" pod "14e66787-45ea-4e62-83f8-e14a0388b915" (UID: "14e66787-45ea-4e62-83f8-e14a0388b915"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:44.218594 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.218538 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14e66787-45ea-4e62-83f8-e14a0388b915-maas-api-tls" pod "14e66787-45ea-4e62-83f8-e14a0388b915" (UID: "14e66787-45ea-4e62-83f8-e14a0388b915"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:44.218594 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.218585 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e66787-45ea-4e62-83f8-e14a0388b915-kube-api-access-tt7dd" pod "14e66787-45ea-4e62-83f8-e14a0388b915" (UID: "14e66787-45ea-4e62-83f8-e14a0388b915"). InnerVolumeSpecName "kube-api-access-tt7dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:44.318457 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.318399 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14e66787-45ea-4e62-83f8-e14a0388b915-maas-api-tls\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:44.318625 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.318469 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt7dd\" (UniqueName: \"kubernetes.io/projected/14e66787-45ea-4e62-83f8-e14a0388b915-kube-api-access-tt7dd\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:44.318625 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.318502 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14e66787-45ea-4e62-83f8-e14a0388b915-metrics-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:44.391541 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.391506 4918 generic.go:356] "Generic (PLEG): container finished" podID="14e66787-45ea-4e62-83f8-e14a0388b915" containerID="adc37e0e1df0f755ccac79a67244d11d4338893f1a82296b1eaf9b814506a480" exitCode=0 Nov 03 20:33:44.392023 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.391565 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" Nov 03 20:33:44.392023 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.391589 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" event={"ID":"14e66787-45ea-4e62-83f8-e14a0388b915","Type":"ContainerDied","Data":"adc37e0e1df0f755ccac79a67244d11d4338893f1a82296b1eaf9b814506a480"} Nov 03 20:33:44.392023 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.391647 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb" event={"ID":"14e66787-45ea-4e62-83f8-e14a0388b915","Type":"ContainerDied","Data":"9381c32fc3c3656afe9e62e1168fb72f19bd930968ebbac25e4144c7276b974f"} Nov 03 20:33:44.392023 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.391671 4918 scope.go:122] "RemoveContainer" containerID="adc37e0e1df0f755ccac79a67244d11d4338893f1a82296b1eaf9b814506a480" Nov 03 20:33:44.408737 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.408716 4918 scope.go:122] "RemoveContainer" containerID="adc37e0e1df0f755ccac79a67244d11d4338893f1a82296b1eaf9b814506a480" Nov 03 20:33:44.409012 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:33:44.408992 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adc37e0e1df0f755ccac79a67244d11d4338893f1a82296b1eaf9b814506a480\": container with ID starting with adc37e0e1df0f755ccac79a67244d11d4338893f1a82296b1eaf9b814506a480 not found: ID does not exist" containerID="adc37e0e1df0f755ccac79a67244d11d4338893f1a82296b1eaf9b814506a480" Nov 03 20:33:44.409097 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.409020 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adc37e0e1df0f755ccac79a67244d11d4338893f1a82296b1eaf9b814506a480"} err="failed to get container status \"adc37e0e1df0f755ccac79a67244d11d4338893f1a82296b1eaf9b814506a480\": rpc error: code = NotFound desc = could not find container \"adc37e0e1df0f755ccac79a67244d11d4338893f1a82296b1eaf9b814506a480\": container with ID starting with adc37e0e1df0f755ccac79a67244d11d4338893f1a82296b1eaf9b814506a480 not found: ID does not exist" Nov 03 20:33:44.422661 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.422638 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb"] Nov 03 20:33:44.433589 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.433565 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b0adda-bfd968475-gc7tb"] Nov 03 20:33:44.837066 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:44.837001 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw" podUID="a96cad21-e85e-4757-9e72-6b350738e230" containerName="payload-processing" containerID="cri-o://9016f16a3d639f1ea2a3a88dac11b95115f749cc37af709ccdc679e35aad4bc4" gracePeriod=25 Nov 03 20:33:45.130242 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.130210 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw" Nov 03 20:33:45.217918 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.217885 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14e66787-45ea-4e62-83f8-e14a0388b915" path="/var/lib/kubelet/pods/14e66787-45ea-4e62-83f8-e14a0388b915/volumes" Nov 03 20:33:45.218520 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.218499 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b00acc2-340d-4fec-bab6-efadbf64e74d" path="/var/lib/kubelet/pods/3b00acc2-340d-4fec-bab6-efadbf64e74d/volumes" Nov 03 20:33:45.232270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.232244 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a96cad21-e85e-4757-9e72-6b350738e230-kube-api-access-txlf6\" (UniqueName: \"kubernetes.io/projected/a96cad21-e85e-4757-9e72-6b350738e230-kube-api-access-txlf6\") pod \"a96cad21-e85e-4757-9e72-6b350738e230\" (UID: \"a96cad21-e85e-4757-9e72-6b350738e230\") " Nov 03 20:33:45.232416 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.232396 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a96cad21-e85e-4757-9e72-6b350738e230-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a96cad21-e85e-4757-9e72-6b350738e230-plugins-config-volume\") pod \"a96cad21-e85e-4757-9e72-6b350738e230\" (UID: \"a96cad21-e85e-4757-9e72-6b350738e230\") " Nov 03 20:33:45.233354 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.233322 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a96cad21-e85e-4757-9e72-6b350738e230-plugins-config-volume" pod "a96cad21-e85e-4757-9e72-6b350738e230" (UID: "a96cad21-e85e-4757-9e72-6b350738e230"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:45.235813 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.235779 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96cad21-e85e-4757-9e72-6b350738e230-kube-api-access-txlf6" pod "a96cad21-e85e-4757-9e72-6b350738e230" (UID: "a96cad21-e85e-4757-9e72-6b350738e230"). InnerVolumeSpecName "kube-api-access-txlf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:45.334126 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.334087 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txlf6\" (UniqueName: \"kubernetes.io/projected/a96cad21-e85e-4757-9e72-6b350738e230-kube-api-access-txlf6\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.334126 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.334114 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a96cad21-e85e-4757-9e72-6b350738e230-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.401015 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.400929 4918 generic.go:356] "Generic (PLEG): container finished" podID="a96cad21-e85e-4757-9e72-6b350738e230" containerID="9016f16a3d639f1ea2a3a88dac11b95115f749cc37af709ccdc679e35aad4bc4" exitCode=0 Nov 03 20:33:45.401015 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.400964 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw" event={"ID":"a96cad21-e85e-4757-9e72-6b350738e230","Type":"ContainerDied","Data":"9016f16a3d639f1ea2a3a88dac11b95115f749cc37af709ccdc679e35aad4bc4"} Nov 03 20:33:45.401508 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.401018 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw" event={"ID":"a96cad21-e85e-4757-9e72-6b350738e230","Type":"ContainerDied","Data":"9e917e6ab4db4234f16e56e9942c5bfca99c08942478a40cfc2851f3806fd749"} Nov 03 20:33:45.401508 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.400974 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw" Nov 03 20:33:45.401508 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.401067 4918 scope.go:122] "RemoveContainer" containerID="9016f16a3d639f1ea2a3a88dac11b95115f749cc37af709ccdc679e35aad4bc4" Nov 03 20:33:45.412434 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.412415 4918 scope.go:122] "RemoveContainer" containerID="9016f16a3d639f1ea2a3a88dac11b95115f749cc37af709ccdc679e35aad4bc4" Nov 03 20:33:45.412710 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:33:45.412688 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9016f16a3d639f1ea2a3a88dac11b95115f749cc37af709ccdc679e35aad4bc4\": container with ID starting with 9016f16a3d639f1ea2a3a88dac11b95115f749cc37af709ccdc679e35aad4bc4 not found: ID does not exist" containerID="9016f16a3d639f1ea2a3a88dac11b95115f749cc37af709ccdc679e35aad4bc4" Nov 03 20:33:45.412778 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.412718 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9016f16a3d639f1ea2a3a88dac11b95115f749cc37af709ccdc679e35aad4bc4"} err="failed to get container status \"9016f16a3d639f1ea2a3a88dac11b95115f749cc37af709ccdc679e35aad4bc4\": rpc error: code = NotFound desc = could not find container \"9016f16a3d639f1ea2a3a88dac11b95115f749cc37af709ccdc679e35aad4bc4\": container with ID starting with 9016f16a3d639f1ea2a3a88dac11b95115f749cc37af709ccdc679e35aad4bc4 not found: ID does not exist" Nov 03 20:33:45.428051 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.428025 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw"] Nov 03 20:33:45.442435 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.442410 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3ef4b9-857df9bc6-6lrjw"] Nov 03 20:33:45.768432 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.768404 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:33:45.820551 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.820519 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:33:45.943649 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.943617 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2461bee-954b-481d-a740-379482984a4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2461bee-954b-481d-a740-379482984a4a-istiod-ca-cert\") pod \"b2461bee-954b-481d-a740-379482984a4a\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " Nov 03 20:33:45.943834 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.943659 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-istio-data\") pod \"b2461bee-954b-481d-a740-379482984a4a\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " Nov 03 20:33:45.943834 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.943687 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-istio-data\") pod \"b6290d9f-ddd4-40d2-809b-110249e01125\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " Nov 03 20:33:45.943957 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.943830 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-workload-socket\") pod \"b2461bee-954b-481d-a740-379482984a4a\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " Nov 03 20:33:45.943957 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.943888 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-istio-envoy\") pod \"b2461bee-954b-481d-a740-379482984a4a\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " Nov 03 20:33:45.943957 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.943919 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-workload-certs\") pod \"b2461bee-954b-481d-a740-379482984a4a\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " Nov 03 20:33:45.943957 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.943945 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-workload-socket\") pod \"b6290d9f-ddd4-40d2-809b-110249e01125\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " Nov 03 20:33:45.944199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.943941 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-istio-data" pod "b6290d9f-ddd4-40d2-809b-110249e01125" (UID: "b6290d9f-ddd4-40d2-809b-110249e01125"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.944199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.943940 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-istio-data" pod "b2461bee-954b-481d-a740-379482984a4a" (UID: "b2461bee-954b-481d-a740-379482984a4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.944199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.943992 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6290d9f-ddd4-40d2-809b-110249e01125-kube-api-access-wfvch\" (UniqueName: \"kubernetes.io/projected/b6290d9f-ddd4-40d2-809b-110249e01125-kube-api-access-wfvch\") pod \"b6290d9f-ddd4-40d2-809b-110249e01125\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " Nov 03 20:33:45.944199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.944039 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-workload-certs\") pod \"b6290d9f-ddd4-40d2-809b-110249e01125\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " Nov 03 20:33:45.944199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.944088 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2461bee-954b-481d-a740-379482984a4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2461bee-954b-481d-a740-379482984a4a-istio-podinfo\") pod \"b2461bee-954b-481d-a740-379482984a4a\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " Nov 03 20:33:45.944199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.944148 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2461bee-954b-481d-a740-379482984a4a-kube-api-access-c7qct\" (UniqueName: \"kubernetes.io/projected/b2461bee-954b-481d-a740-379482984a4a-kube-api-access-c7qct\") pod \"b2461bee-954b-481d-a740-379482984a4a\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " Nov 03 20:33:45.944531 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.944112 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2461bee-954b-481d-a740-379482984a4a-istiod-ca-cert" pod "b2461bee-954b-481d-a740-379482984a4a" (UID: "b2461bee-954b-481d-a740-379482984a4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:45.944531 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.944321 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2461bee-954b-481d-a740-379482984a4a-istio-token\" (UniqueName: \"kubernetes.io/projected/b2461bee-954b-481d-a740-379482984a4a-istio-token\") pod \"b2461bee-954b-481d-a740-379482984a4a\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " Nov 03 20:33:45.944531 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.944356 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-workload-certs" pod "b2461bee-954b-481d-a740-379482984a4a" (UID: "b2461bee-954b-481d-a740-379482984a4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.944531 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.944379 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6290d9f-ddd4-40d2-809b-110249e01125-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6290d9f-ddd4-40d2-809b-110249e01125-istiod-ca-cert\") pod \"b6290d9f-ddd4-40d2-809b-110249e01125\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " Nov 03 20:33:45.944531 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.944465 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-workload-socket" pod "b6290d9f-ddd4-40d2-809b-110249e01125" (UID: "b6290d9f-ddd4-40d2-809b-110249e01125"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.944531 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.944474 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6290d9f-ddd4-40d2-809b-110249e01125-istio-token\" (UniqueName: \"kubernetes.io/projected/b6290d9f-ddd4-40d2-809b-110249e01125-istio-token\") pod \"b6290d9f-ddd4-40d2-809b-110249e01125\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " Nov 03 20:33:45.944828 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.944540 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6290d9f-ddd4-40d2-809b-110249e01125-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6290d9f-ddd4-40d2-809b-110249e01125-istio-podinfo\") pod \"b6290d9f-ddd4-40d2-809b-110249e01125\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " Nov 03 20:33:45.944828 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.944621 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-istio-envoy\") pod \"b6290d9f-ddd4-40d2-809b-110249e01125\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " Nov 03 20:33:45.944828 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.944715 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-credential-socket\") pod \"b2461bee-954b-481d-a740-379482984a4a\" (UID: \"b2461bee-954b-481d-a740-379482984a4a\") " Nov 03 20:33:45.944828 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.944751 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-credential-socket\") pod \"b6290d9f-ddd4-40d2-809b-110249e01125\" (UID: \"b6290d9f-ddd4-40d2-809b-110249e01125\") " Nov 03 20:33:45.945534 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.945314 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-workload-certs" pod "b6290d9f-ddd4-40d2-809b-110249e01125" (UID: "b6290d9f-ddd4-40d2-809b-110249e01125"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.945652 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.945564 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6290d9f-ddd4-40d2-809b-110249e01125-istiod-ca-cert" pod "b6290d9f-ddd4-40d2-809b-110249e01125" (UID: "b6290d9f-ddd4-40d2-809b-110249e01125"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:45.945652 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.945576 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-credential-socket" pod "b2461bee-954b-481d-a740-379482984a4a" (UID: "b2461bee-954b-481d-a740-379482984a4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.945780 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.945760 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-credential-socket" pod "b6290d9f-ddd4-40d2-809b-110249e01125" (UID: "b6290d9f-ddd4-40d2-809b-110249e01125"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.945927 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.945888 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-workload-socket" pod "b2461bee-954b-481d-a740-379482984a4a" (UID: "b2461bee-954b-481d-a740-379482984a4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.946706 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.946527 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.946706 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.946555 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.946706 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.946572 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2461bee-954b-481d-a740-379482984a4a-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.946706 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.946587 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.946706 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.946602 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.946706 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.946618 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.946706 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.946632 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.946706 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.946647 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.946706 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.946661 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.946706 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.946676 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6290d9f-ddd4-40d2-809b-110249e01125-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.947514 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.947313 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2461bee-954b-481d-a740-379482984a4a-istio-podinfo" pod "b2461bee-954b-481d-a740-379482984a4a" (UID: "b2461bee-954b-481d-a740-379482984a4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:45.949432 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.948842 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-istio-envoy" pod "b6290d9f-ddd4-40d2-809b-110249e01125" (UID: "b6290d9f-ddd4-40d2-809b-110249e01125"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.949432 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.948870 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6290d9f-ddd4-40d2-809b-110249e01125-istio-token" pod "b6290d9f-ddd4-40d2-809b-110249e01125" (UID: "b6290d9f-ddd4-40d2-809b-110249e01125"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:45.949432 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.948936 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2461bee-954b-481d-a740-379482984a4a-istio-token" pod "b2461bee-954b-481d-a740-379482984a4a" (UID: "b2461bee-954b-481d-a740-379482984a4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:45.949432 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.948962 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6290d9f-ddd4-40d2-809b-110249e01125-kube-api-access-wfvch" pod "b6290d9f-ddd4-40d2-809b-110249e01125" (UID: "b6290d9f-ddd4-40d2-809b-110249e01125"). InnerVolumeSpecName "kube-api-access-wfvch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:45.949432 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.949168 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-istio-envoy" pod "b2461bee-954b-481d-a740-379482984a4a" (UID: "b2461bee-954b-481d-a740-379482984a4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.949762 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.949738 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2461bee-954b-481d-a740-379482984a4a-kube-api-access-c7qct" pod "b2461bee-954b-481d-a740-379482984a4a" (UID: "b2461bee-954b-481d-a740-379482984a4a"). InnerVolumeSpecName "kube-api-access-c7qct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:45.949908 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:45.949892 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6290d9f-ddd4-40d2-809b-110249e01125-istio-podinfo" pod "b6290d9f-ddd4-40d2-809b-110249e01125" (UID: "b6290d9f-ddd4-40d2-809b-110249e01125"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:46.047920 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.047849 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2461bee-954b-481d-a740-379482984a4a-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.047920 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.047877 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfvch\" (UniqueName: \"kubernetes.io/projected/b6290d9f-ddd4-40d2-809b-110249e01125-kube-api-access-wfvch\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.047920 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.047889 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2461bee-954b-481d-a740-379482984a4a-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.047920 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.047899 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7qct\" (UniqueName: \"kubernetes.io/projected/b2461bee-954b-481d-a740-379482984a4a-kube-api-access-c7qct\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.047920 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.047907 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2461bee-954b-481d-a740-379482984a4a-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.047920 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.047916 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6290d9f-ddd4-40d2-809b-110249e01125-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.047920 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.047925 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6290d9f-ddd4-40d2-809b-110249e01125-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.048271 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.047933 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6290d9f-ddd4-40d2-809b-110249e01125-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.460523 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.460344 4918 generic.go:356] "Generic (PLEG): container finished" podID="b2461bee-954b-481d-a740-379482984a4a" containerID="d63bf761de529013fca363edfa148446ae91f5a1eab8e6c77b46173101569dcf" exitCode=0 Nov 03 20:33:46.461277 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.461244 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" event={"ID":"b2461bee-954b-481d-a740-379482984a4a","Type":"ContainerDied","Data":"d63bf761de529013fca363edfa148446ae91f5a1eab8e6c77b46173101569dcf"} Nov 03 20:33:46.461490 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.461469 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" event={"ID":"b2461bee-954b-481d-a740-379482984a4a","Type":"ContainerDied","Data":"df14d7c114a19bd4758508752d54fe139f118fa8fa472a48f14b1f6efd9eee4f"} Nov 03 20:33:46.461614 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.461600 4918 scope.go:122] "RemoveContainer" containerID="d63bf761de529013fca363edfa148446ae91f5a1eab8e6c77b46173101569dcf" Nov 03 20:33:46.461822 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.461807 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc" Nov 03 20:33:46.476330 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.476054 4918 generic.go:356] "Generic (PLEG): container finished" podID="b6290d9f-ddd4-40d2-809b-110249e01125" containerID="3f90c4195c6cc9050988f91e71616e07936195cfa707d38e8fda88008ad2f111" exitCode=0 Nov 03 20:33:46.476330 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.476191 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" event={"ID":"b6290d9f-ddd4-40d2-809b-110249e01125","Type":"ContainerDied","Data":"3f90c4195c6cc9050988f91e71616e07936195cfa707d38e8fda88008ad2f111"} Nov 03 20:33:46.476330 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.476266 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" event={"ID":"b6290d9f-ddd4-40d2-809b-110249e01125","Type":"ContainerDied","Data":"c44472434e556fe5aad83a5d4ce2e2686c3d8747f723642dd85706062b931b3c"} Nov 03 20:33:46.476824 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.476644 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj" Nov 03 20:33:46.499674 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.499650 4918 scope.go:122] "RemoveContainer" containerID="d63bf761de529013fca363edfa148446ae91f5a1eab8e6c77b46173101569dcf" Nov 03 20:33:46.500030 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:33:46.500004 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d63bf761de529013fca363edfa148446ae91f5a1eab8e6c77b46173101569dcf\": container with ID starting with d63bf761de529013fca363edfa148446ae91f5a1eab8e6c77b46173101569dcf not found: ID does not exist" containerID="d63bf761de529013fca363edfa148446ae91f5a1eab8e6c77b46173101569dcf" Nov 03 20:33:46.500075 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.500039 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d63bf761de529013fca363edfa148446ae91f5a1eab8e6c77b46173101569dcf"} err="failed to get container status \"d63bf761de529013fca363edfa148446ae91f5a1eab8e6c77b46173101569dcf\": rpc error: code = NotFound desc = could not find container \"d63bf761de529013fca363edfa148446ae91f5a1eab8e6c77b46173101569dcf\": container with ID starting with d63bf761de529013fca363edfa148446ae91f5a1eab8e6c77b46173101569dcf not found: ID does not exist" Nov 03 20:33:46.500075 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.500057 4918 scope.go:122] "RemoveContainer" containerID="3f90c4195c6cc9050988f91e71616e07936195cfa707d38e8fda88008ad2f111" Nov 03 20:33:46.515661 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.515631 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc"] Nov 03 20:33:46.520399 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.520372 4918 scope.go:122] "RemoveContainer" containerID="3f90c4195c6cc9050988f91e71616e07936195cfa707d38e8fda88008ad2f111" Nov 03 20:33:46.524766 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:33:46.524730 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f90c4195c6cc9050988f91e71616e07936195cfa707d38e8fda88008ad2f111\": container with ID starting with 3f90c4195c6cc9050988f91e71616e07936195cfa707d38e8fda88008ad2f111 not found: ID does not exist" containerID="3f90c4195c6cc9050988f91e71616e07936195cfa707d38e8fda88008ad2f111" Nov 03 20:33:46.524884 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.524776 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f90c4195c6cc9050988f91e71616e07936195cfa707d38e8fda88008ad2f111"} err="failed to get container status \"3f90c4195c6cc9050988f91e71616e07936195cfa707d38e8fda88008ad2f111\": rpc error: code = NotFound desc = could not find container \"3f90c4195c6cc9050988f91e71616e07936195cfa707d38e8fda88008ad2f111\": container with ID starting with 3f90c4195c6cc9050988f91e71616e07936195cfa707d38e8fda88008ad2f111 not found: ID does not exist" Nov 03 20:33:46.527473 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.527446 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wszsc"] Nov 03 20:33:46.539375 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.539339 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj"] Nov 03 20:33:46.542850 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:46.542815 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sdtjj"] Nov 03 20:33:47.216831 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.216796 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a96cad21-e85e-4757-9e72-6b350738e230" path="/var/lib/kubelet/pods/a96cad21-e85e-4757-9e72-6b350738e230/volumes" Nov 03 20:33:47.217280 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.217264 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2461bee-954b-481d-a740-379482984a4a" path="/var/lib/kubelet/pods/b2461bee-954b-481d-a740-379482984a4a/volumes" Nov 03 20:33:47.217705 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.217692 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6290d9f-ddd4-40d2-809b-110249e01125" path="/var/lib/kubelet/pods/b6290d9f-ddd4-40d2-809b-110249e01125/volumes" Nov 03 20:33:47.487511 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.487064 4918 generic.go:356] "Generic (PLEG): container finished" podID="ea246681-cf7a-40b9-97d1-9c1d2594c3ec" containerID="5ee2fb2cb5bab41990dacff40ca3ad55ce1d6623450c2264e1a870d6db532581" exitCode=0 Nov 03 20:33:47.487511 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.487138 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" event={"ID":"ea246681-cf7a-40b9-97d1-9c1d2594c3ec","Type":"ContainerDied","Data":"5ee2fb2cb5bab41990dacff40ca3ad55ce1d6623450c2264e1a870d6db532581"} Nov 03 20:33:47.596979 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.596947 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:33:47.685809 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.685768 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-kube-api-access-vmnbm\" (UniqueName: \"kubernetes.io/projected/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-kube-api-access-vmnbm\") pod \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " Nov 03 20:33:47.685982 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.685833 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-data\") pod \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " Nov 03 20:33:47.685982 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.685903 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-envoy\") pod \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " Nov 03 20:33:47.685982 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.685966 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-podinfo\") pod \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " Nov 03 20:33:47.686167 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.685996 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-workload-certs\") pod \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " Nov 03 20:33:47.686167 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.686039 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-workload-socket\") pod \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " Nov 03 20:33:47.686167 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.686066 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istiod-ca-cert\") pod \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " Nov 03 20:33:47.686167 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.686135 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-credential-socket\") pod \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " Nov 03 20:33:47.686409 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.686223 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-data" pod "ea246681-cf7a-40b9-97d1-9c1d2594c3ec" (UID: "ea246681-cf7a-40b9-97d1-9c1d2594c3ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:47.686409 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.686393 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-token\" (UniqueName: \"kubernetes.io/projected/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-token\") pod \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\" (UID: \"ea246681-cf7a-40b9-97d1-9c1d2594c3ec\") " Nov 03 20:33:47.686511 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.686455 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-workload-certs" pod "ea246681-cf7a-40b9-97d1-9c1d2594c3ec" (UID: "ea246681-cf7a-40b9-97d1-9c1d2594c3ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:47.686511 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.686479 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-workload-socket" pod "ea246681-cf7a-40b9-97d1-9c1d2594c3ec" (UID: "ea246681-cf7a-40b9-97d1-9c1d2594c3ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:47.686668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.686631 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-credential-socket" pod "ea246681-cf7a-40b9-97d1-9c1d2594c3ec" (UID: "ea246681-cf7a-40b9-97d1-9c1d2594c3ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:47.687040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.687010 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istiod-ca-cert" pod "ea246681-cf7a-40b9-97d1-9c1d2594c3ec" (UID: "ea246681-cf7a-40b9-97d1-9c1d2594c3ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:47.687150 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.687135 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.687243 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.687156 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.687243 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.687172 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.687243 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.687205 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.687243 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.687219 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.689043 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.689014 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-envoy" pod "ea246681-cf7a-40b9-97d1-9c1d2594c3ec" (UID: "ea246681-cf7a-40b9-97d1-9c1d2594c3ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:47.689144 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.689076 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-kube-api-access-vmnbm" pod "ea246681-cf7a-40b9-97d1-9c1d2594c3ec" (UID: "ea246681-cf7a-40b9-97d1-9c1d2594c3ec"). InnerVolumeSpecName "kube-api-access-vmnbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:47.689144 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.689120 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-podinfo" pod "ea246681-cf7a-40b9-97d1-9c1d2594c3ec" (UID: "ea246681-cf7a-40b9-97d1-9c1d2594c3ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:47.689523 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.689496 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-token" pod "ea246681-cf7a-40b9-97d1-9c1d2594c3ec" (UID: "ea246681-cf7a-40b9-97d1-9c1d2594c3ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:47.788785 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.788715 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.788785 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.788742 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmnbm\" (UniqueName: \"kubernetes.io/projected/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-kube-api-access-vmnbm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.788785 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.788753 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.788785 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:47.788763 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea246681-cf7a-40b9-97d1-9c1d2594c3ec-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:48.497943 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:48.497910 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" event={"ID":"ea246681-cf7a-40b9-97d1-9c1d2594c3ec","Type":"ContainerDied","Data":"3c345e9f9a80b5bd5b42eec4ac418ec33135ebc035ccf9774d865b4e14e4f103"} Nov 03 20:33:48.498439 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:48.497969 4918 scope.go:122] "RemoveContainer" containerID="5ee2fb2cb5bab41990dacff40ca3ad55ce1d6623450c2264e1a870d6db532581" Nov 03 20:33:48.498439 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:48.497903 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk" Nov 03 20:33:48.523821 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:48.523794 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk"] Nov 03 20:33:48.528398 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:48.528374 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-nbswk"] Nov 03 20:33:49.216253 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:49.216222 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea246681-cf7a-40b9-97d1-9c1d2594c3ec" path="/var/lib/kubelet/pods/ea246681-cf7a-40b9-97d1-9c1d2594c3ec/volumes" Nov 03 20:33:50.229663 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:50.229625 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb"] Nov 03 20:33:50.230167 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:50.229856 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" podUID="2f8f2527-d745-4a36-b656-594d782b0db3" containerName="istio-proxy" containerID="cri-o://00811c572e166823a0f507735808690d98923f3209146aad7f69a4cf90e7a4fc" gracePeriod=30 Nov 03 20:33:51.106903 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:51.106864 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp"] Nov 03 20:33:51.123892 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:51.123858 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw"] Nov 03 20:33:53.582201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:53.582156 4918 patch_prober.go:29] interesting pod/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:33:53.582616 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:53.582233 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" podUID="2f8f2527-d745-4a36-b656-594d782b0db3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:33:55.531128 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.531101 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:33:55.593296 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.593258 4918 generic.go:356] "Generic (PLEG): container finished" podID="2f8f2527-d745-4a36-b656-594d782b0db3" containerID="00811c572e166823a0f507735808690d98923f3209146aad7f69a4cf90e7a4fc" exitCode=0 Nov 03 20:33:55.593812 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.593777 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" event={"ID":"2f8f2527-d745-4a36-b656-594d782b0db3","Type":"ContainerDied","Data":"00811c572e166823a0f507735808690d98923f3209146aad7f69a4cf90e7a4fc"} Nov 03 20:33:55.593978 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.593854 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" event={"ID":"2f8f2527-d745-4a36-b656-594d782b0db3","Type":"ContainerDied","Data":"408dc6d2e17028e59584851cdcc06eb8ac6fa9ff79bd4efce648caafb36898a4"} Nov 03 20:33:55.593978 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.593888 4918 scope.go:122] "RemoveContainer" containerID="00811c572e166823a0f507735808690d98923f3209146aad7f69a4cf90e7a4fc" Nov 03 20:33:55.594093 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.594056 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb" Nov 03 20:33:55.618885 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.618856 4918 scope.go:122] "RemoveContainer" containerID="3b9d321a30241bc19fdee48a378432cead8677df92bad7a99e0cef03a356fcb1" Nov 03 20:33:55.633538 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.633498 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f8f2527-d745-4a36-b656-594d782b0db3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8f2527-d745-4a36-b656-594d782b0db3-istio-podinfo\") pod \"2f8f2527-d745-4a36-b656-594d782b0db3\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " Nov 03 20:33:55.633727 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.633592 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-istio-data\") pod \"2f8f2527-d745-4a36-b656-594d782b0db3\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " Nov 03 20:33:55.633727 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.633654 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-istio-envoy\") pod \"2f8f2527-d745-4a36-b656-594d782b0db3\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " Nov 03 20:33:55.633897 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.633744 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-workload-socket\") pod \"2f8f2527-d745-4a36-b656-594d782b0db3\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " Nov 03 20:33:55.633897 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.633844 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-credential-socket\") pod \"2f8f2527-d745-4a36-b656-594d782b0db3\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " Nov 03 20:33:55.634012 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.633894 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f8f2527-d745-4a36-b656-594d782b0db3-kube-api-access-6zvw4\" (UniqueName: \"kubernetes.io/projected/2f8f2527-d745-4a36-b656-594d782b0db3-kube-api-access-6zvw4\") pod \"2f8f2527-d745-4a36-b656-594d782b0db3\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " Nov 03 20:33:55.634012 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.633931 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-workload-certs\") pod \"2f8f2527-d745-4a36-b656-594d782b0db3\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " Nov 03 20:33:55.634012 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.633966 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f8f2527-d745-4a36-b656-594d782b0db3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8f2527-d745-4a36-b656-594d782b0db3-istiod-ca-cert\") pod \"2f8f2527-d745-4a36-b656-594d782b0db3\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " Nov 03 20:33:55.634170 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.634116 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f8f2527-d745-4a36-b656-594d782b0db3-istio-token\" (UniqueName: \"kubernetes.io/projected/2f8f2527-d745-4a36-b656-594d782b0db3-istio-token\") pod \"2f8f2527-d745-4a36-b656-594d782b0db3\" (UID: \"2f8f2527-d745-4a36-b656-594d782b0db3\") " Nov 03 20:33:55.634581 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.634550 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-workload-socket" pod "2f8f2527-d745-4a36-b656-594d782b0db3" (UID: "2f8f2527-d745-4a36-b656-594d782b0db3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:55.637150 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.636912 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8f2527-d745-4a36-b656-594d782b0db3-istiod-ca-cert" pod "2f8f2527-d745-4a36-b656-594d782b0db3" (UID: "2f8f2527-d745-4a36-b656-594d782b0db3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:55.637367 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.637336 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-credential-socket" pod "2f8f2527-d745-4a36-b656-594d782b0db3" (UID: "2f8f2527-d745-4a36-b656-594d782b0db3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:55.637565 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.637543 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-istio-data" pod "2f8f2527-d745-4a36-b656-594d782b0db3" (UID: "2f8f2527-d745-4a36-b656-594d782b0db3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:55.637972 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.637947 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-workload-certs" pod "2f8f2527-d745-4a36-b656-594d782b0db3" (UID: "2f8f2527-d745-4a36-b656-594d782b0db3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:55.641166 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.641140 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:55.648334 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.648313 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:55.648444 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.648339 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8f2527-d745-4a36-b656-594d782b0db3-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:55.648444 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.648355 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:55.648444 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.648115 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-istio-envoy" pod "2f8f2527-d745-4a36-b656-594d782b0db3" (UID: "2f8f2527-d745-4a36-b656-594d782b0db3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:55.648444 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.648370 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:55.648444 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.648262 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8f2527-d745-4a36-b656-594d782b0db3-istio-token" pod "2f8f2527-d745-4a36-b656-594d782b0db3" (UID: "2f8f2527-d745-4a36-b656-594d782b0db3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:55.648648 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.648621 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8f2527-d745-4a36-b656-594d782b0db3-kube-api-access-6zvw4" pod "2f8f2527-d745-4a36-b656-594d782b0db3" (UID: "2f8f2527-d745-4a36-b656-594d782b0db3"). InnerVolumeSpecName "kube-api-access-6zvw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:55.653822 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.652336 4918 scope.go:122] "RemoveContainer" containerID="00811c572e166823a0f507735808690d98923f3209146aad7f69a4cf90e7a4fc" Nov 03 20:33:55.653822 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:33:55.653629 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00811c572e166823a0f507735808690d98923f3209146aad7f69a4cf90e7a4fc\": container with ID starting with 00811c572e166823a0f507735808690d98923f3209146aad7f69a4cf90e7a4fc not found: ID does not exist" containerID="00811c572e166823a0f507735808690d98923f3209146aad7f69a4cf90e7a4fc" Nov 03 20:33:55.653822 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.653674 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00811c572e166823a0f507735808690d98923f3209146aad7f69a4cf90e7a4fc"} err="failed to get container status \"00811c572e166823a0f507735808690d98923f3209146aad7f69a4cf90e7a4fc\": rpc error: code = NotFound desc = could not find container \"00811c572e166823a0f507735808690d98923f3209146aad7f69a4cf90e7a4fc\": container with ID starting with 00811c572e166823a0f507735808690d98923f3209146aad7f69a4cf90e7a4fc not found: ID does not exist" Nov 03 20:33:55.653822 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.653704 4918 scope.go:122] "RemoveContainer" containerID="3b9d321a30241bc19fdee48a378432cead8677df92bad7a99e0cef03a356fcb1" Nov 03 20:33:55.654149 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:33:55.654087 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9d321a30241bc19fdee48a378432cead8677df92bad7a99e0cef03a356fcb1\": container with ID starting with 3b9d321a30241bc19fdee48a378432cead8677df92bad7a99e0cef03a356fcb1 not found: ID does not exist" containerID="3b9d321a30241bc19fdee48a378432cead8677df92bad7a99e0cef03a356fcb1" Nov 03 20:33:55.654149 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.654125 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9d321a30241bc19fdee48a378432cead8677df92bad7a99e0cef03a356fcb1"} err="failed to get container status \"3b9d321a30241bc19fdee48a378432cead8677df92bad7a99e0cef03a356fcb1\": rpc error: code = NotFound desc = could not find container \"3b9d321a30241bc19fdee48a378432cead8677df92bad7a99e0cef03a356fcb1\": container with ID starting with 3b9d321a30241bc19fdee48a378432cead8677df92bad7a99e0cef03a356fcb1 not found: ID does not exist" Nov 03 20:33:55.654579 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.654554 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f8f2527-d745-4a36-b656-594d782b0db3-istio-podinfo" pod "2f8f2527-d745-4a36-b656-594d782b0db3" (UID: "2f8f2527-d745-4a36-b656-594d782b0db3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:55.750270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.750238 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8f2527-d745-4a36-b656-594d782b0db3-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:55.750424 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.750296 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zvw4\" (UniqueName: \"kubernetes.io/projected/2f8f2527-d745-4a36-b656-594d782b0db3-kube-api-access-6zvw4\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:55.750424 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.750309 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8f2527-d745-4a36-b656-594d782b0db3-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:55.750424 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.750318 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8f2527-d745-4a36-b656-594d782b0db3-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:55.920255 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.920223 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb"] Nov 03 20:33:55.924656 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:55.924629 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3ef4b9-openshift-default-bb7c7fdf9-qwgsb"] Nov 03 20:33:56.107487 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.107442 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp" podUID="d270ff2c-258f-4c66-b50e-1bbb2494816d" containerName="payload-processing" containerID="cri-o://4b9bcba9e92025c1b16871c76abd5b474d0ecc233a8f84964a273b76085b9002" gracePeriod=25 Nov 03 20:33:56.124604 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.124572 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw" podUID="2ea5be43-cacb-48ac-89bb-ca19bdc86522" containerName="payload-pre-processing" containerID="cri-o://1d76475cf7316fc3ccd41796daa6204fee0fe1a6d3e13eb966302f90ef71fb68" gracePeriod=25 Nov 03 20:33:56.442782 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.442707 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp" Nov 03 20:33:56.457232 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.457089 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw" Nov 03 20:33:56.563861 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.563830 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ea5be43-cacb-48ac-89bb-ca19bdc86522-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ea5be43-cacb-48ac-89bb-ca19bdc86522-plugins-config-volume\") pod \"2ea5be43-cacb-48ac-89bb-ca19bdc86522\" (UID: \"2ea5be43-cacb-48ac-89bb-ca19bdc86522\") " Nov 03 20:33:56.564355 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.563897 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d270ff2c-258f-4c66-b50e-1bbb2494816d-kube-api-access-gwf9m\" (UniqueName: \"kubernetes.io/projected/d270ff2c-258f-4c66-b50e-1bbb2494816d-kube-api-access-gwf9m\") pod \"d270ff2c-258f-4c66-b50e-1bbb2494816d\" (UID: \"d270ff2c-258f-4c66-b50e-1bbb2494816d\") " Nov 03 20:33:56.564355 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.564050 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d270ff2c-258f-4c66-b50e-1bbb2494816d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d270ff2c-258f-4c66-b50e-1bbb2494816d-plugins-config-volume\") pod \"d270ff2c-258f-4c66-b50e-1bbb2494816d\" (UID: \"d270ff2c-258f-4c66-b50e-1bbb2494816d\") " Nov 03 20:33:56.564355 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.564119 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea5be43-cacb-48ac-89bb-ca19bdc86522-kube-api-access-26xck\" (UniqueName: \"kubernetes.io/projected/2ea5be43-cacb-48ac-89bb-ca19bdc86522-kube-api-access-26xck\") pod \"2ea5be43-cacb-48ac-89bb-ca19bdc86522\" (UID: \"2ea5be43-cacb-48ac-89bb-ca19bdc86522\") " Nov 03 20:33:56.564355 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.564300 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea5be43-cacb-48ac-89bb-ca19bdc86522-plugins-config-volume" pod "2ea5be43-cacb-48ac-89bb-ca19bdc86522" (UID: "2ea5be43-cacb-48ac-89bb-ca19bdc86522"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:56.564592 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.564513 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d270ff2c-258f-4c66-b50e-1bbb2494816d-plugins-config-volume" pod "d270ff2c-258f-4c66-b50e-1bbb2494816d" (UID: "d270ff2c-258f-4c66-b50e-1bbb2494816d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:56.564980 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.564956 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ea5be43-cacb-48ac-89bb-ca19bdc86522-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:56.565059 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.564983 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d270ff2c-258f-4c66-b50e-1bbb2494816d-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:56.566132 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.566102 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d270ff2c-258f-4c66-b50e-1bbb2494816d-kube-api-access-gwf9m" pod "d270ff2c-258f-4c66-b50e-1bbb2494816d" (UID: "d270ff2c-258f-4c66-b50e-1bbb2494816d"). InnerVolumeSpecName "kube-api-access-gwf9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:56.566234 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.566166 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea5be43-cacb-48ac-89bb-ca19bdc86522-kube-api-access-26xck" pod "2ea5be43-cacb-48ac-89bb-ca19bdc86522" (UID: "2ea5be43-cacb-48ac-89bb-ca19bdc86522"). InnerVolumeSpecName "kube-api-access-26xck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:56.602287 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.602255 4918 generic.go:356] "Generic (PLEG): container finished" podID="d270ff2c-258f-4c66-b50e-1bbb2494816d" containerID="4b9bcba9e92025c1b16871c76abd5b474d0ecc233a8f84964a273b76085b9002" exitCode=0 Nov 03 20:33:56.602403 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.602311 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp" Nov 03 20:33:56.602403 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.602329 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp" event={"ID":"d270ff2c-258f-4c66-b50e-1bbb2494816d","Type":"ContainerDied","Data":"4b9bcba9e92025c1b16871c76abd5b474d0ecc233a8f84964a273b76085b9002"} Nov 03 20:33:56.602403 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.602394 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp" event={"ID":"d270ff2c-258f-4c66-b50e-1bbb2494816d","Type":"ContainerDied","Data":"2f1f513a3cbd18376ad1187566498fc4f7e8dd8dd986590a99def89cc5057972"} Nov 03 20:33:56.602566 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.602433 4918 scope.go:122] "RemoveContainer" containerID="4b9bcba9e92025c1b16871c76abd5b474d0ecc233a8f84964a273b76085b9002" Nov 03 20:33:56.603870 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.603852 4918 generic.go:356] "Generic (PLEG): container finished" podID="2ea5be43-cacb-48ac-89bb-ca19bdc86522" containerID="1d76475cf7316fc3ccd41796daa6204fee0fe1a6d3e13eb966302f90ef71fb68" exitCode=0 Nov 03 20:33:56.603962 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.603914 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw" Nov 03 20:33:56.604033 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.603958 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw" event={"ID":"2ea5be43-cacb-48ac-89bb-ca19bdc86522","Type":"ContainerDied","Data":"1d76475cf7316fc3ccd41796daa6204fee0fe1a6d3e13eb966302f90ef71fb68"} Nov 03 20:33:56.604033 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.603998 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw" event={"ID":"2ea5be43-cacb-48ac-89bb-ca19bdc86522","Type":"ContainerDied","Data":"6e3cf8c6a6db0fd225f7bdf807a7db7d7ff38bb2c10a5d8dc3a1bedc4dbfa582"} Nov 03 20:33:56.613555 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.613489 4918 scope.go:122] "RemoveContainer" containerID="4b9bcba9e92025c1b16871c76abd5b474d0ecc233a8f84964a273b76085b9002" Nov 03 20:33:56.613874 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:33:56.613844 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b9bcba9e92025c1b16871c76abd5b474d0ecc233a8f84964a273b76085b9002\": container with ID starting with 4b9bcba9e92025c1b16871c76abd5b474d0ecc233a8f84964a273b76085b9002 not found: ID does not exist" containerID="4b9bcba9e92025c1b16871c76abd5b474d0ecc233a8f84964a273b76085b9002" Nov 03 20:33:56.613972 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.613883 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9bcba9e92025c1b16871c76abd5b474d0ecc233a8f84964a273b76085b9002"} err="failed to get container status \"4b9bcba9e92025c1b16871c76abd5b474d0ecc233a8f84964a273b76085b9002\": rpc error: code = NotFound desc = could not find container \"4b9bcba9e92025c1b16871c76abd5b474d0ecc233a8f84964a273b76085b9002\": container with ID starting with 4b9bcba9e92025c1b16871c76abd5b474d0ecc233a8f84964a273b76085b9002 not found: ID does not exist" Nov 03 20:33:56.613972 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.613905 4918 scope.go:122] "RemoveContainer" containerID="1d76475cf7316fc3ccd41796daa6204fee0fe1a6d3e13eb966302f90ef71fb68" Nov 03 20:33:56.623894 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.623878 4918 scope.go:122] "RemoveContainer" containerID="1d76475cf7316fc3ccd41796daa6204fee0fe1a6d3e13eb966302f90ef71fb68" Nov 03 20:33:56.624186 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:33:56.624152 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d76475cf7316fc3ccd41796daa6204fee0fe1a6d3e13eb966302f90ef71fb68\": container with ID starting with 1d76475cf7316fc3ccd41796daa6204fee0fe1a6d3e13eb966302f90ef71fb68 not found: ID does not exist" containerID="1d76475cf7316fc3ccd41796daa6204fee0fe1a6d3e13eb966302f90ef71fb68" Nov 03 20:33:56.624251 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.624200 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d76475cf7316fc3ccd41796daa6204fee0fe1a6d3e13eb966302f90ef71fb68"} err="failed to get container status \"1d76475cf7316fc3ccd41796daa6204fee0fe1a6d3e13eb966302f90ef71fb68\": rpc error: code = NotFound desc = could not find container \"1d76475cf7316fc3ccd41796daa6204fee0fe1a6d3e13eb966302f90ef71fb68\": container with ID starting with 1d76475cf7316fc3ccd41796daa6204fee0fe1a6d3e13eb966302f90ef71fb68 not found: ID does not exist" Nov 03 20:33:56.633289 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.633269 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw"] Nov 03 20:33:56.639130 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.639108 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b35c30-67ccd5fc87-s4mcw"] Nov 03 20:33:56.649731 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.649712 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp"] Nov 03 20:33:56.654531 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.654511 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b35c30-6c9756b5b-q87pp"] Nov 03 20:33:56.666763 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.666742 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwf9m\" (UniqueName: \"kubernetes.io/projected/d270ff2c-258f-4c66-b50e-1bbb2494816d-kube-api-access-gwf9m\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:56.666840 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:56.666766 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26xck\" (UniqueName: \"kubernetes.io/projected/2ea5be43-cacb-48ac-89bb-ca19bdc86522-kube-api-access-26xck\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:57.239046 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:57.238930 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea5be43-cacb-48ac-89bb-ca19bdc86522" path="/var/lib/kubelet/pods/2ea5be43-cacb-48ac-89bb-ca19bdc86522/volumes" Nov 03 20:33:57.240480 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:57.240451 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f8f2527-d745-4a36-b656-594d782b0db3" path="/var/lib/kubelet/pods/2f8f2527-d745-4a36-b656-594d782b0db3/volumes" Nov 03 20:33:57.241172 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:33:57.241146 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d270ff2c-258f-4c66-b50e-1bbb2494816d" path="/var/lib/kubelet/pods/d270ff2c-258f-4c66-b50e-1bbb2494816d/volumes" Nov 03 20:33:58.209738 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:33:58.209705 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:34:02.089672 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.089554 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z"] Nov 03 20:34:02.090307 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.089916 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" podUID="d986da59-41d5-4c2e-8832-e7e79e5e2b19" containerName="maas-api" containerID="cri-o://4fd3f8bd2a2d34539b9840dc1a84c0c5209ad1881c138f96a14362766e1d520a" gracePeriod=30 Nov 03 20:34:02.492815 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.492783 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" Nov 03 20:34:02.536815 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.536772 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d986da59-41d5-4c2e-8832-e7e79e5e2b19-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d986da59-41d5-4c2e-8832-e7e79e5e2b19-maas-api-tls\") pod \"d986da59-41d5-4c2e-8832-e7e79e5e2b19\" (UID: \"d986da59-41d5-4c2e-8832-e7e79e5e2b19\") " Nov 03 20:34:02.537006 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.536917 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d986da59-41d5-4c2e-8832-e7e79e5e2b19-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d986da59-41d5-4c2e-8832-e7e79e5e2b19-openshift-service-ca\") pod \"d986da59-41d5-4c2e-8832-e7e79e5e2b19\" (UID: \"d986da59-41d5-4c2e-8832-e7e79e5e2b19\") " Nov 03 20:34:02.537006 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.536973 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d986da59-41d5-4c2e-8832-e7e79e5e2b19-kube-api-access-k8vrs\" (UniqueName: \"kubernetes.io/projected/d986da59-41d5-4c2e-8832-e7e79e5e2b19-kube-api-access-k8vrs\") pod \"d986da59-41d5-4c2e-8832-e7e79e5e2b19\" (UID: \"d986da59-41d5-4c2e-8832-e7e79e5e2b19\") " Nov 03 20:34:02.537135 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.537017 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d986da59-41d5-4c2e-8832-e7e79e5e2b19-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d986da59-41d5-4c2e-8832-e7e79e5e2b19-metrics-certs\") pod \"d986da59-41d5-4c2e-8832-e7e79e5e2b19\" (UID: \"d986da59-41d5-4c2e-8832-e7e79e5e2b19\") " Nov 03 20:34:02.538290 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.538262 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d986da59-41d5-4c2e-8832-e7e79e5e2b19-openshift-service-ca" pod "d986da59-41d5-4c2e-8832-e7e79e5e2b19" (UID: "d986da59-41d5-4c2e-8832-e7e79e5e2b19"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:02.539914 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.539889 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d986da59-41d5-4c2e-8832-e7e79e5e2b19-metrics-certs" pod "d986da59-41d5-4c2e-8832-e7e79e5e2b19" (UID: "d986da59-41d5-4c2e-8832-e7e79e5e2b19"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:02.540303 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.540280 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d986da59-41d5-4c2e-8832-e7e79e5e2b19-kube-api-access-k8vrs" pod "d986da59-41d5-4c2e-8832-e7e79e5e2b19" (UID: "d986da59-41d5-4c2e-8832-e7e79e5e2b19"). InnerVolumeSpecName "kube-api-access-k8vrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:02.540392 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.540331 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d986da59-41d5-4c2e-8832-e7e79e5e2b19-maas-api-tls" pod "d986da59-41d5-4c2e-8832-e7e79e5e2b19" (UID: "d986da59-41d5-4c2e-8832-e7e79e5e2b19"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:02.640763 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.640575 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d986da59-41d5-4c2e-8832-e7e79e5e2b19-openshift-service-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:02.644611 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.644555 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8vrs\" (UniqueName: \"kubernetes.io/projected/d986da59-41d5-4c2e-8832-e7e79e5e2b19-kube-api-access-k8vrs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:02.653452 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.653408 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d986da59-41d5-4c2e-8832-e7e79e5e2b19-metrics-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:02.653452 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.653442 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d986da59-41d5-4c2e-8832-e7e79e5e2b19-maas-api-tls\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:02.673249 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.673211 4918 generic.go:356] "Generic (PLEG): container finished" podID="d986da59-41d5-4c2e-8832-e7e79e5e2b19" containerID="4fd3f8bd2a2d34539b9840dc1a84c0c5209ad1881c138f96a14362766e1d520a" exitCode=0 Nov 03 20:34:02.673452 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.673267 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" Nov 03 20:34:02.673452 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.673274 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" event={"ID":"d986da59-41d5-4c2e-8832-e7e79e5e2b19","Type":"ContainerDied","Data":"4fd3f8bd2a2d34539b9840dc1a84c0c5209ad1881c138f96a14362766e1d520a"} Nov 03 20:34:02.673452 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.673332 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z" event={"ID":"d986da59-41d5-4c2e-8832-e7e79e5e2b19","Type":"ContainerDied","Data":"6979394d3887bf3eb79843f06afdaa8322865f84fc541ee429ee346619e6abd5"} Nov 03 20:34:02.673452 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.673354 4918 scope.go:122] "RemoveContainer" containerID="4fd3f8bd2a2d34539b9840dc1a84c0c5209ad1881c138f96a14362766e1d520a" Nov 03 20:34:02.684510 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.684490 4918 scope.go:122] "RemoveContainer" containerID="4fd3f8bd2a2d34539b9840dc1a84c0c5209ad1881c138f96a14362766e1d520a" Nov 03 20:34:02.684785 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:34:02.684764 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd3f8bd2a2d34539b9840dc1a84c0c5209ad1881c138f96a14362766e1d520a\": container with ID starting with 4fd3f8bd2a2d34539b9840dc1a84c0c5209ad1881c138f96a14362766e1d520a not found: ID does not exist" containerID="4fd3f8bd2a2d34539b9840dc1a84c0c5209ad1881c138f96a14362766e1d520a" Nov 03 20:34:02.684830 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.684793 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd3f8bd2a2d34539b9840dc1a84c0c5209ad1881c138f96a14362766e1d520a"} err="failed to get container status \"4fd3f8bd2a2d34539b9840dc1a84c0c5209ad1881c138f96a14362766e1d520a\": rpc error: code = NotFound desc = could not find container \"4fd3f8bd2a2d34539b9840dc1a84c0c5209ad1881c138f96a14362766e1d520a\": container with ID starting with 4fd3f8bd2a2d34539b9840dc1a84c0c5209ad1881c138f96a14362766e1d520a not found: ID does not exist" Nov 03 20:34:02.697682 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.697658 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z"] Nov 03 20:34:02.703355 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:02.703333 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ba8f5b-56c88766b6-thb2z"] Nov 03 20:34:03.216619 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:03.216581 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d986da59-41d5-4c2e-8832-e7e79e5e2b19" path="/var/lib/kubelet/pods/d986da59-41d5-4c2e-8832-e7e79e5e2b19/volumes" Nov 03 20:34:08.421468 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:08.417535 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn"] Nov 03 20:34:08.421468 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:08.417868 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" podUID="9fb1ff44-4b90-47da-a252-d5771f160330" containerName="istio-proxy" containerID="cri-o://e40f2bef5ce3d5d70256258930c28e79d505eb14701844c3b3e3f3e854e7110e" gracePeriod=30 Nov 03 20:34:10.393373 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:10.393336 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7"] Nov 03 20:34:10.393858 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:10.393588 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" podUID="1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" containerName="istio-proxy" containerID="cri-o://fdc3ac4d958ebed156e75a66d521365ebc3f4da98f8b258ce638b0378b7fbb17" gracePeriod=30 Nov 03 20:34:12.668157 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:12.668110 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m"] Nov 03 20:34:12.680146 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:12.680113 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7"] Nov 03 20:34:13.762784 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:13.762740 4918 generic.go:356] "Generic (PLEG): container finished" podID="9fb1ff44-4b90-47da-a252-d5771f160330" containerID="e40f2bef5ce3d5d70256258930c28e79d505eb14701844c3b3e3f3e854e7110e" exitCode=0 Nov 03 20:34:13.764619 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:13.762949 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" event={"ID":"9fb1ff44-4b90-47da-a252-d5771f160330","Type":"ContainerDied","Data":"e40f2bef5ce3d5d70256258930c28e79d505eb14701844c3b3e3f3e854e7110e"} Nov 03 20:34:13.764619 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:13.763015 4918 scope.go:122] "RemoveContainer" containerID="82bc338730c7f8bd7bd15db32eb5bf81f54b601a7e1da65350bcef80e6b9a38f" Nov 03 20:34:13.845220 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:13.845141 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:34:13.988045 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:13.988010 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb1ff44-4b90-47da-a252-d5771f160330-kube-api-access-s72xw\" (UniqueName: \"kubernetes.io/projected/9fb1ff44-4b90-47da-a252-d5771f160330-kube-api-access-s72xw\") pod \"9fb1ff44-4b90-47da-a252-d5771f160330\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " Nov 03 20:34:13.988747 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:13.988722 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fb1ff44-4b90-47da-a252-d5771f160330-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fb1ff44-4b90-47da-a252-d5771f160330-istiod-ca-cert\") pod \"9fb1ff44-4b90-47da-a252-d5771f160330\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " Nov 03 20:34:13.989418 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:13.989299 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb1ff44-4b90-47da-a252-d5771f160330-istiod-ca-cert" pod "9fb1ff44-4b90-47da-a252-d5771f160330" (UID: "9fb1ff44-4b90-47da-a252-d5771f160330"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:13.989551 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:13.989371 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-credential-socket\") pod \"9fb1ff44-4b90-47da-a252-d5771f160330\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " Nov 03 20:34:13.991780 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:13.989695 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-credential-socket" pod "9fb1ff44-4b90-47da-a252-d5771f160330" (UID: "9fb1ff44-4b90-47da-a252-d5771f160330"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:13.994216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:13.992863 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-istio-data\") pod \"9fb1ff44-4b90-47da-a252-d5771f160330\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " Nov 03 20:34:13.994216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:13.992969 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-workload-socket\") pod \"9fb1ff44-4b90-47da-a252-d5771f160330\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " Nov 03 20:34:13.994216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:13.993013 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-workload-certs\") pod \"9fb1ff44-4b90-47da-a252-d5771f160330\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " Nov 03 20:34:13.994216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:13.993060 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-istio-envoy\") pod \"9fb1ff44-4b90-47da-a252-d5771f160330\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " Nov 03 20:34:13.994216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:13.993138 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb1ff44-4b90-47da-a252-d5771f160330-istio-token\" (UniqueName: \"kubernetes.io/projected/9fb1ff44-4b90-47da-a252-d5771f160330-istio-token\") pod \"9fb1ff44-4b90-47da-a252-d5771f160330\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " Nov 03 20:34:13.994216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:13.993199 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fb1ff44-4b90-47da-a252-d5771f160330-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fb1ff44-4b90-47da-a252-d5771f160330-istio-podinfo\") pod \"9fb1ff44-4b90-47da-a252-d5771f160330\" (UID: \"9fb1ff44-4b90-47da-a252-d5771f160330\") " Nov 03 20:34:13.994216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:13.993807 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fb1ff44-4b90-47da-a252-d5771f160330-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:13.994216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:13.993826 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:13.994877 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:13.994852 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-workload-certs" pod "9fb1ff44-4b90-47da-a252-d5771f160330" (UID: "9fb1ff44-4b90-47da-a252-d5771f160330"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:13.995002 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:13.994906 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-workload-socket" pod "9fb1ff44-4b90-47da-a252-d5771f160330" (UID: "9fb1ff44-4b90-47da-a252-d5771f160330"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:13.995160 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:13.994953 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-istio-data" pod "9fb1ff44-4b90-47da-a252-d5771f160330" (UID: "9fb1ff44-4b90-47da-a252-d5771f160330"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:14.007559 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:14.007503 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb1ff44-4b90-47da-a252-d5771f160330-istio-token" pod "9fb1ff44-4b90-47da-a252-d5771f160330" (UID: "9fb1ff44-4b90-47da-a252-d5771f160330"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:14.011267 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:14.011237 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb1ff44-4b90-47da-a252-d5771f160330-kube-api-access-s72xw" pod "9fb1ff44-4b90-47da-a252-d5771f160330" (UID: "9fb1ff44-4b90-47da-a252-d5771f160330"). InnerVolumeSpecName "kube-api-access-s72xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:14.013600 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:14.013573 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-istio-envoy" pod "9fb1ff44-4b90-47da-a252-d5771f160330" (UID: "9fb1ff44-4b90-47da-a252-d5771f160330"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:14.014953 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:14.014926 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fb1ff44-4b90-47da-a252-d5771f160330-istio-podinfo" pod "9fb1ff44-4b90-47da-a252-d5771f160330" (UID: "9fb1ff44-4b90-47da-a252-d5771f160330"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:14.095034 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:14.094992 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:14.095034 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:14.095026 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:14.095034 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:14.095037 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:14.095337 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:14.095046 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fb1ff44-4b90-47da-a252-d5771f160330-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:14.095337 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:14.095056 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fb1ff44-4b90-47da-a252-d5771f160330-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:14.095337 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:14.095065 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fb1ff44-4b90-47da-a252-d5771f160330-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:14.095337 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:14.095076 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s72xw\" (UniqueName: \"kubernetes.io/projected/9fb1ff44-4b90-47da-a252-d5771f160330-kube-api-access-s72xw\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:14.770969 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:14.770939 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" Nov 03 20:34:14.771430 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:14.771036 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn" event={"ID":"9fb1ff44-4b90-47da-a252-d5771f160330","Type":"ContainerDied","Data":"e532b728c34063c75374c40c512fefe61df0f52e47eefdec9df79a1687501f34"} Nov 03 20:34:14.771430 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:14.771084 4918 scope.go:122] "RemoveContainer" containerID="e40f2bef5ce3d5d70256258930c28e79d505eb14701844c3b3e3f3e854e7110e" Nov 03 20:34:14.798492 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:14.798464 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn"] Nov 03 20:34:14.801544 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:14.801519 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b35c30-openshift-default-7bb98df9c5-kwwwn"] Nov 03 20:34:15.215522 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.215485 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fb1ff44-4b90-47da-a252-d5771f160330" path="/var/lib/kubelet/pods/9fb1ff44-4b90-47da-a252-d5771f160330/volumes" Nov 03 20:34:15.683152 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.683124 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:34:15.780132 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.780050 4918 generic.go:356] "Generic (PLEG): container finished" podID="1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" containerID="fdc3ac4d958ebed156e75a66d521365ebc3f4da98f8b258ce638b0378b7fbb17" exitCode=0 Nov 03 20:34:15.780132 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.780106 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" Nov 03 20:34:15.780631 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.780113 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" event={"ID":"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7","Type":"ContainerDied","Data":"fdc3ac4d958ebed156e75a66d521365ebc3f4da98f8b258ce638b0378b7fbb17"} Nov 03 20:34:15.780631 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.780246 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7" event={"ID":"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7","Type":"ContainerDied","Data":"ee3b3557fc99b35ccbe0e30baf3e68fcd58ec9e1c49dc8e050325bef95e685d8"} Nov 03 20:34:15.780631 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.780271 4918 scope.go:122] "RemoveContainer" containerID="fdc3ac4d958ebed156e75a66d521365ebc3f4da98f8b258ce638b0378b7fbb17" Nov 03 20:34:15.790958 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.790940 4918 scope.go:122] "RemoveContainer" containerID="7320646a9c45f1fbc4bae0aed37c5df44d18947d80ea929fc253520e10799ba2" Nov 03 20:34:15.800625 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.800607 4918 scope.go:122] "RemoveContainer" containerID="fdc3ac4d958ebed156e75a66d521365ebc3f4da98f8b258ce638b0378b7fbb17" Nov 03 20:34:15.800901 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:34:15.800881 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc3ac4d958ebed156e75a66d521365ebc3f4da98f8b258ce638b0378b7fbb17\": container with ID starting with fdc3ac4d958ebed156e75a66d521365ebc3f4da98f8b258ce638b0378b7fbb17 not found: ID does not exist" containerID="fdc3ac4d958ebed156e75a66d521365ebc3f4da98f8b258ce638b0378b7fbb17" Nov 03 20:34:15.800951 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.800911 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc3ac4d958ebed156e75a66d521365ebc3f4da98f8b258ce638b0378b7fbb17"} err="failed to get container status \"fdc3ac4d958ebed156e75a66d521365ebc3f4da98f8b258ce638b0378b7fbb17\": rpc error: code = NotFound desc = could not find container \"fdc3ac4d958ebed156e75a66d521365ebc3f4da98f8b258ce638b0378b7fbb17\": container with ID starting with fdc3ac4d958ebed156e75a66d521365ebc3f4da98f8b258ce638b0378b7fbb17 not found: ID does not exist" Nov 03 20:34:15.800951 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.800930 4918 scope.go:122] "RemoveContainer" containerID="7320646a9c45f1fbc4bae0aed37c5df44d18947d80ea929fc253520e10799ba2" Nov 03 20:34:15.801161 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:34:15.801141 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7320646a9c45f1fbc4bae0aed37c5df44d18947d80ea929fc253520e10799ba2\": container with ID starting with 7320646a9c45f1fbc4bae0aed37c5df44d18947d80ea929fc253520e10799ba2 not found: ID does not exist" containerID="7320646a9c45f1fbc4bae0aed37c5df44d18947d80ea929fc253520e10799ba2" Nov 03 20:34:15.801243 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.801171 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7320646a9c45f1fbc4bae0aed37c5df44d18947d80ea929fc253520e10799ba2"} err="failed to get container status \"7320646a9c45f1fbc4bae0aed37c5df44d18947d80ea929fc253520e10799ba2\": rpc error: code = NotFound desc = could not find container \"7320646a9c45f1fbc4bae0aed37c5df44d18947d80ea929fc253520e10799ba2\": container with ID starting with 7320646a9c45f1fbc4bae0aed37c5df44d18947d80ea929fc253520e10799ba2 not found: ID does not exist" Nov 03 20:34:15.818801 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.818778 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-data\") pod \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " Nov 03 20:34:15.818900 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.818846 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-envoy\") pod \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " Nov 03 20:34:15.818965 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.818896 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-credential-socket\") pod \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " Nov 03 20:34:15.818965 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.818935 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-kube-api-access-cgz9p\" (UniqueName: \"kubernetes.io/projected/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-kube-api-access-cgz9p\") pod \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " Nov 03 20:34:15.819071 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.818976 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-token\" (UniqueName: \"kubernetes.io/projected/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-token\") pod \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " Nov 03 20:34:15.819071 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.819001 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-data" pod "1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" (UID: "1d98bfc5-35c3-4925-a3d1-696dd0f90dd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:15.819168 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.819099 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-workload-certs\") pod \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " Nov 03 20:34:15.819168 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.819153 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-workload-socket\") pod \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " Nov 03 20:34:15.819294 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.819232 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-credential-socket" pod "1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" (UID: "1d98bfc5-35c3-4925-a3d1-696dd0f90dd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:15.819294 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.819276 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-podinfo\") pod \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " Nov 03 20:34:15.819401 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.819324 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istiod-ca-cert\") pod \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\" (UID: \"1d98bfc5-35c3-4925-a3d1-696dd0f90dd7\") " Nov 03 20:34:15.819401 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.819350 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-workload-certs" pod "1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" (UID: "1d98bfc5-35c3-4925-a3d1-696dd0f90dd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:15.819497 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.819476 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-workload-socket" pod "1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" (UID: "1d98bfc5-35c3-4925-a3d1-696dd0f90dd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:15.819701 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.819677 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istiod-ca-cert" pod "1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" (UID: "1d98bfc5-35c3-4925-a3d1-696dd0f90dd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:15.819905 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.819887 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:15.819905 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.819906 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:15.820042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.819916 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:15.820042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.819925 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:15.820042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.819935 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:15.821106 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.821078 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-envoy" pod "1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" (UID: "1d98bfc5-35c3-4925-a3d1-696dd0f90dd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:15.821428 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.821401 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-token" pod "1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" (UID: "1d98bfc5-35c3-4925-a3d1-696dd0f90dd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:15.821517 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.821489 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-podinfo" pod "1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" (UID: "1d98bfc5-35c3-4925-a3d1-696dd0f90dd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:15.821835 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.821816 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-kube-api-access-cgz9p" pod "1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" (UID: "1d98bfc5-35c3-4925-a3d1-696dd0f90dd7"). InnerVolumeSpecName "kube-api-access-cgz9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:15.921152 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.921112 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:15.921152 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.921144 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgz9p\" (UniqueName: \"kubernetes.io/projected/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-kube-api-access-cgz9p\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:15.921152 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.921159 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:15.921464 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:15.921171 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:16.115297 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:16.115256 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7"] Nov 03 20:34:16.118457 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:16.118428 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ba8f5b-openshift-default-85b675648c-z55b7"] Nov 03 20:34:17.232348 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:17.227946 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" path="/var/lib/kubelet/pods/1d98bfc5-35c3-4925-a3d1-696dd0f90dd7/volumes" Nov 03 20:34:17.668802 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:17.668761 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m" podUID="1a6a964b-7ff6-4c06-b46f-499c03cfd744" containerName="payload-processing" containerID="cri-o://38276d246d6865295bd91b2dc9479a1bb8d96dfc418956e60005919a2b69ef25" gracePeriod=25 Nov 03 20:34:17.681323 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:17.681289 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7" podUID="4e6db5de-2704-45a9-8e69-a78d3cf1fb5a" containerName="payload-pre-processing" containerID="cri-o://fc5023a86df4184899628bba8bc97c4d89b28f11f6ed56d8394eb1cf990d9b12" gracePeriod=25 Nov 03 20:34:17.796348 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:17.796311 4918 generic.go:356] "Generic (PLEG): container finished" podID="1a6a964b-7ff6-4c06-b46f-499c03cfd744" containerID="38276d246d6865295bd91b2dc9479a1bb8d96dfc418956e60005919a2b69ef25" exitCode=0 Nov 03 20:34:17.796465 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:17.796380 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m" event={"ID":"1a6a964b-7ff6-4c06-b46f-499c03cfd744","Type":"ContainerDied","Data":"38276d246d6865295bd91b2dc9479a1bb8d96dfc418956e60005919a2b69ef25"} Nov 03 20:34:17.989162 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:17.989131 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m" Nov 03 20:34:18.011928 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.011899 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7" Nov 03 20:34:18.074639 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.074601 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a6a964b-7ff6-4c06-b46f-499c03cfd744-kube-api-access-kr24j\" (UniqueName: \"kubernetes.io/projected/1a6a964b-7ff6-4c06-b46f-499c03cfd744-kube-api-access-kr24j\") pod \"1a6a964b-7ff6-4c06-b46f-499c03cfd744\" (UID: \"1a6a964b-7ff6-4c06-b46f-499c03cfd744\") " Nov 03 20:34:18.074834 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.074681 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e6db5de-2704-45a9-8e69-a78d3cf1fb5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e6db5de-2704-45a9-8e69-a78d3cf1fb5a-plugins-config-volume\") pod \"4e6db5de-2704-45a9-8e69-a78d3cf1fb5a\" (UID: \"4e6db5de-2704-45a9-8e69-a78d3cf1fb5a\") " Nov 03 20:34:18.074908 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.074842 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e6db5de-2704-45a9-8e69-a78d3cf1fb5a-kube-api-access-g5ms2\" (UniqueName: \"kubernetes.io/projected/4e6db5de-2704-45a9-8e69-a78d3cf1fb5a-kube-api-access-g5ms2\") pod \"4e6db5de-2704-45a9-8e69-a78d3cf1fb5a\" (UID: \"4e6db5de-2704-45a9-8e69-a78d3cf1fb5a\") " Nov 03 20:34:18.074908 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.074898 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a6a964b-7ff6-4c06-b46f-499c03cfd744-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6a964b-7ff6-4c06-b46f-499c03cfd744-plugins-config-volume\") pod \"1a6a964b-7ff6-4c06-b46f-499c03cfd744\" (UID: \"1a6a964b-7ff6-4c06-b46f-499c03cfd744\") " Nov 03 20:34:18.075236 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.075088 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6db5de-2704-45a9-8e69-a78d3cf1fb5a-plugins-config-volume" pod "4e6db5de-2704-45a9-8e69-a78d3cf1fb5a" (UID: "4e6db5de-2704-45a9-8e69-a78d3cf1fb5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:18.075364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.075278 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a6a964b-7ff6-4c06-b46f-499c03cfd744-plugins-config-volume" pod "1a6a964b-7ff6-4c06-b46f-499c03cfd744" (UID: "1a6a964b-7ff6-4c06-b46f-499c03cfd744"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:18.075462 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.075438 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6a964b-7ff6-4c06-b46f-499c03cfd744-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:18.075582 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.075462 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e6db5de-2704-45a9-8e69-a78d3cf1fb5a-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:18.076989 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.076964 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6db5de-2704-45a9-8e69-a78d3cf1fb5a-kube-api-access-g5ms2" pod "4e6db5de-2704-45a9-8e69-a78d3cf1fb5a" (UID: "4e6db5de-2704-45a9-8e69-a78d3cf1fb5a"). InnerVolumeSpecName "kube-api-access-g5ms2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:18.077160 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.077136 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6a964b-7ff6-4c06-b46f-499c03cfd744-kube-api-access-kr24j" pod "1a6a964b-7ff6-4c06-b46f-499c03cfd744" (UID: "1a6a964b-7ff6-4c06-b46f-499c03cfd744"). InnerVolumeSpecName "kube-api-access-kr24j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:18.177219 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.177170 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr24j\" (UniqueName: \"kubernetes.io/projected/1a6a964b-7ff6-4c06-b46f-499c03cfd744-kube-api-access-kr24j\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:18.177219 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.177215 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5ms2\" (UniqueName: \"kubernetes.io/projected/4e6db5de-2704-45a9-8e69-a78d3cf1fb5a-kube-api-access-g5ms2\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:18.804608 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.804580 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m" Nov 03 20:34:18.805070 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.804577 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m" event={"ID":"1a6a964b-7ff6-4c06-b46f-499c03cfd744","Type":"ContainerDied","Data":"dba7759d3695d994cd51acd86e36c639e839cf361ff8e1cb4919fdec0529e63e"} Nov 03 20:34:18.805070 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.804718 4918 scope.go:122] "RemoveContainer" containerID="38276d246d6865295bd91b2dc9479a1bb8d96dfc418956e60005919a2b69ef25" Nov 03 20:34:18.806089 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.806064 4918 generic.go:356] "Generic (PLEG): container finished" podID="4e6db5de-2704-45a9-8e69-a78d3cf1fb5a" containerID="fc5023a86df4184899628bba8bc97c4d89b28f11f6ed56d8394eb1cf990d9b12" exitCode=0 Nov 03 20:34:18.806166 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.806101 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7" Nov 03 20:34:18.806166 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.806148 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7" event={"ID":"4e6db5de-2704-45a9-8e69-a78d3cf1fb5a","Type":"ContainerDied","Data":"fc5023a86df4184899628bba8bc97c4d89b28f11f6ed56d8394eb1cf990d9b12"} Nov 03 20:34:18.806287 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.806252 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7" event={"ID":"4e6db5de-2704-45a9-8e69-a78d3cf1fb5a","Type":"ContainerDied","Data":"d87dcc2f4090169ccee44353bf93e4cca79ea68429cd695b567569019368adc3"} Nov 03 20:34:18.816326 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.816306 4918 scope.go:122] "RemoveContainer" containerID="fc5023a86df4184899628bba8bc97c4d89b28f11f6ed56d8394eb1cf990d9b12" Nov 03 20:34:18.828601 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.828573 4918 scope.go:122] "RemoveContainer" containerID="fc5023a86df4184899628bba8bc97c4d89b28f11f6ed56d8394eb1cf990d9b12" Nov 03 20:34:18.828939 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:34:18.828914 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc5023a86df4184899628bba8bc97c4d89b28f11f6ed56d8394eb1cf990d9b12\": container with ID starting with fc5023a86df4184899628bba8bc97c4d89b28f11f6ed56d8394eb1cf990d9b12 not found: ID does not exist" containerID="fc5023a86df4184899628bba8bc97c4d89b28f11f6ed56d8394eb1cf990d9b12" Nov 03 20:34:18.831452 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.828959 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc5023a86df4184899628bba8bc97c4d89b28f11f6ed56d8394eb1cf990d9b12"} err="failed to get container status \"fc5023a86df4184899628bba8bc97c4d89b28f11f6ed56d8394eb1cf990d9b12\": rpc error: code = NotFound desc = could not find container \"fc5023a86df4184899628bba8bc97c4d89b28f11f6ed56d8394eb1cf990d9b12\": container with ID starting with fc5023a86df4184899628bba8bc97c4d89b28f11f6ed56d8394eb1cf990d9b12 not found: ID does not exist" Nov 03 20:34:18.835225 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.835197 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m"] Nov 03 20:34:18.838767 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.838743 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3e9176-d6677b4bb-txv7m"] Nov 03 20:34:18.847190 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.847154 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7"] Nov 03 20:34:18.851942 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:18.851918 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3e9176-67b7f447b7-tkxd7"] Nov 03 20:34:19.192918 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:19.190815 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs"] Nov 03 20:34:19.198319 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:19.198291 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl"] Nov 03 20:34:19.222219 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:19.221096 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a6a964b-7ff6-4c06-b46f-499c03cfd744" path="/var/lib/kubelet/pods/1a6a964b-7ff6-4c06-b46f-499c03cfd744/volumes" Nov 03 20:34:19.222219 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:19.221560 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e6db5de-2704-45a9-8e69-a78d3cf1fb5a" path="/var/lib/kubelet/pods/4e6db5de-2704-45a9-8e69-a78d3cf1fb5a/volumes" Nov 03 20:34:22.347903 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.347867 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8"] Nov 03 20:34:22.348427 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.348154 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" podUID="4c77123e-9c94-4299-a7e0-1759315d24fd" containerName="maas-api" containerID="cri-o://699b538d19fb2fa483c35e211d4b92d0d8aa2ff2a0440402a5eb25c6ee39962e" gracePeriod=30 Nov 03 20:34:22.480591 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.480558 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg"] Nov 03 20:34:22.480821 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.480779 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" podUID="c7a163ac-b7b9-460f-a0e5-73faac7e3f48" containerName="maas-api" containerID="cri-o://a5496cbb9192787365551b802157912e61ebd891e143d70fe059f69deb757b39" gracePeriod=30 Nov 03 20:34:22.644458 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.644427 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" Nov 03 20:34:22.760200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.756638 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c77123e-9c94-4299-a7e0-1759315d24fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c77123e-9c94-4299-a7e0-1759315d24fd-metrics-certs\") pod \"4c77123e-9c94-4299-a7e0-1759315d24fd\" (UID: \"4c77123e-9c94-4299-a7e0-1759315d24fd\") " Nov 03 20:34:22.760200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.756779 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c77123e-9c94-4299-a7e0-1759315d24fd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c77123e-9c94-4299-a7e0-1759315d24fd-openshift-service-ca\") pod \"4c77123e-9c94-4299-a7e0-1759315d24fd\" (UID: \"4c77123e-9c94-4299-a7e0-1759315d24fd\") " Nov 03 20:34:22.760200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.756930 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c77123e-9c94-4299-a7e0-1759315d24fd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c77123e-9c94-4299-a7e0-1759315d24fd-maas-api-tls\") pod \"4c77123e-9c94-4299-a7e0-1759315d24fd\" (UID: \"4c77123e-9c94-4299-a7e0-1759315d24fd\") " Nov 03 20:34:22.760200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.757011 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c77123e-9c94-4299-a7e0-1759315d24fd-kube-api-access-9lzrg\" (UniqueName: \"kubernetes.io/projected/4c77123e-9c94-4299-a7e0-1759315d24fd-kube-api-access-9lzrg\") pod \"4c77123e-9c94-4299-a7e0-1759315d24fd\" (UID: \"4c77123e-9c94-4299-a7e0-1759315d24fd\") " Nov 03 20:34:22.775208 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.767411 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c77123e-9c94-4299-a7e0-1759315d24fd-openshift-service-ca" pod "4c77123e-9c94-4299-a7e0-1759315d24fd" (UID: "4c77123e-9c94-4299-a7e0-1759315d24fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:22.793932 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.790671 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c77123e-9c94-4299-a7e0-1759315d24fd-openshift-service-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:22.797830 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.794376 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c77123e-9c94-4299-a7e0-1759315d24fd-kube-api-access-9lzrg" pod "4c77123e-9c94-4299-a7e0-1759315d24fd" (UID: "4c77123e-9c94-4299-a7e0-1759315d24fd"). InnerVolumeSpecName "kube-api-access-9lzrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:22.819540 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.819120 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c77123e-9c94-4299-a7e0-1759315d24fd-metrics-certs" pod "4c77123e-9c94-4299-a7e0-1759315d24fd" (UID: "4c77123e-9c94-4299-a7e0-1759315d24fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:22.825385 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.823373 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c77123e-9c94-4299-a7e0-1759315d24fd-maas-api-tls" pod "4c77123e-9c94-4299-a7e0-1759315d24fd" (UID: "4c77123e-9c94-4299-a7e0-1759315d24fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:22.836560 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.836524 4918 generic.go:356] "Generic (PLEG): container finished" podID="4c77123e-9c94-4299-a7e0-1759315d24fd" containerID="699b538d19fb2fa483c35e211d4b92d0d8aa2ff2a0440402a5eb25c6ee39962e" exitCode=0 Nov 03 20:34:22.836711 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.836576 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" Nov 03 20:34:22.836711 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.836613 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" event={"ID":"4c77123e-9c94-4299-a7e0-1759315d24fd","Type":"ContainerDied","Data":"699b538d19fb2fa483c35e211d4b92d0d8aa2ff2a0440402a5eb25c6ee39962e"} Nov 03 20:34:22.836711 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.836683 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8" event={"ID":"4c77123e-9c94-4299-a7e0-1759315d24fd","Type":"ContainerDied","Data":"165712305d6bc46bd4d709d7c1be005da969127a04c7ff8e2136994c552db69b"} Nov 03 20:34:22.836853 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.836713 4918 scope.go:122] "RemoveContainer" containerID="699b538d19fb2fa483c35e211d4b92d0d8aa2ff2a0440402a5eb25c6ee39962e" Nov 03 20:34:22.838675 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.838651 4918 generic.go:356] "Generic (PLEG): container finished" podID="c7a163ac-b7b9-460f-a0e5-73faac7e3f48" containerID="a5496cbb9192787365551b802157912e61ebd891e143d70fe059f69deb757b39" exitCode=0 Nov 03 20:34:22.838828 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.838713 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" event={"ID":"c7a163ac-b7b9-460f-a0e5-73faac7e3f48","Type":"ContainerDied","Data":"a5496cbb9192787365551b802157912e61ebd891e143d70fe059f69deb757b39"} Nov 03 20:34:22.848639 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.848619 4918 scope.go:122] "RemoveContainer" containerID="699b538d19fb2fa483c35e211d4b92d0d8aa2ff2a0440402a5eb25c6ee39962e" Nov 03 20:34:22.848857 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.848842 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" Nov 03 20:34:22.848926 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:34:22.848874 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"699b538d19fb2fa483c35e211d4b92d0d8aa2ff2a0440402a5eb25c6ee39962e\": container with ID starting with 699b538d19fb2fa483c35e211d4b92d0d8aa2ff2a0440402a5eb25c6ee39962e not found: ID does not exist" containerID="699b538d19fb2fa483c35e211d4b92d0d8aa2ff2a0440402a5eb25c6ee39962e" Nov 03 20:34:22.848926 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.848898 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"699b538d19fb2fa483c35e211d4b92d0d8aa2ff2a0440402a5eb25c6ee39962e"} err="failed to get container status \"699b538d19fb2fa483c35e211d4b92d0d8aa2ff2a0440402a5eb25c6ee39962e\": rpc error: code = NotFound desc = could not find container \"699b538d19fb2fa483c35e211d4b92d0d8aa2ff2a0440402a5eb25c6ee39962e\": container with ID starting with 699b538d19fb2fa483c35e211d4b92d0d8aa2ff2a0440402a5eb25c6ee39962e not found: ID does not exist" Nov 03 20:34:22.862519 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.862457 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8"] Nov 03 20:34:22.867358 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.867338 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3e9176-5f856bcc55-rnqh8"] Nov 03 20:34:22.892772 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.892739 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c77123e-9c94-4299-a7e0-1759315d24fd-maas-api-tls\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:22.892913 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.892776 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lzrg\" (UniqueName: \"kubernetes.io/projected/4c77123e-9c94-4299-a7e0-1759315d24fd-kube-api-access-9lzrg\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:22.892913 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.892793 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c77123e-9c94-4299-a7e0-1759315d24fd-metrics-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:22.998920 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.998880 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-openshift-service-ca\") pod \"c7a163ac-b7b9-460f-a0e5-73faac7e3f48\" (UID: \"c7a163ac-b7b9-460f-a0e5-73faac7e3f48\") " Nov 03 20:34:22.999346 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.999310 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-metrics-certs\") pod \"c7a163ac-b7b9-460f-a0e5-73faac7e3f48\" (UID: \"c7a163ac-b7b9-460f-a0e5-73faac7e3f48\") " Nov 03 20:34:22.999591 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.999567 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-maas-api-tls\") pod \"c7a163ac-b7b9-460f-a0e5-73faac7e3f48\" (UID: \"c7a163ac-b7b9-460f-a0e5-73faac7e3f48\") " Nov 03 20:34:22.999726 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:22.999686 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-kube-api-access-9mql7\" (UniqueName: \"kubernetes.io/projected/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-kube-api-access-9mql7\") pod \"c7a163ac-b7b9-460f-a0e5-73faac7e3f48\" (UID: \"c7a163ac-b7b9-460f-a0e5-73faac7e3f48\") " Nov 03 20:34:23.000751 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:23.000721 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-openshift-service-ca" pod "c7a163ac-b7b9-460f-a0e5-73faac7e3f48" (UID: "c7a163ac-b7b9-460f-a0e5-73faac7e3f48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:23.002600 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:23.002575 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-openshift-service-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.006159 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:23.003567 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-maas-api-tls" pod "c7a163ac-b7b9-460f-a0e5-73faac7e3f48" (UID: "c7a163ac-b7b9-460f-a0e5-73faac7e3f48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:23.006159 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:23.003772 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-metrics-certs" pod "c7a163ac-b7b9-460f-a0e5-73faac7e3f48" (UID: "c7a163ac-b7b9-460f-a0e5-73faac7e3f48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:23.006159 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:23.005457 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-kube-api-access-9mql7" pod "c7a163ac-b7b9-460f-a0e5-73faac7e3f48" (UID: "c7a163ac-b7b9-460f-a0e5-73faac7e3f48"). InnerVolumeSpecName "kube-api-access-9mql7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:23.104277 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:23.104237 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-maas-api-tls\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.104277 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:23.104271 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mql7\" (UniqueName: \"kubernetes.io/projected/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-kube-api-access-9mql7\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.104277 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:23.104282 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a163ac-b7b9-460f-a0e5-73faac7e3f48-metrics-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.215501 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:23.215416 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c77123e-9c94-4299-a7e0-1759315d24fd" path="/var/lib/kubelet/pods/4c77123e-9c94-4299-a7e0-1759315d24fd/volumes" Nov 03 20:34:23.866099 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:23.866070 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" Nov 03 20:34:23.866596 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:23.866062 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg" event={"ID":"c7a163ac-b7b9-460f-a0e5-73faac7e3f48","Type":"ContainerDied","Data":"ecc15594203910b0787fce888b721fbc750d3fe4ba5b00206ac5b4cc65895c29"} Nov 03 20:34:23.866596 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:23.866246 4918 scope.go:122] "RemoveContainer" containerID="a5496cbb9192787365551b802157912e61ebd891e143d70fe059f69deb757b39" Nov 03 20:34:23.890585 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:23.890552 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg"] Nov 03 20:34:23.894438 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:23.894415 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ee4c78-b76b8c7cf-njqkg"] Nov 03 20:34:24.191838 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.191729 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs" podUID="ef65466e-4f77-454d-9089-a7b0c636b563" containerName="payload-pre-processing" containerID="cri-o://da6a647af7b41ab083ffd5d3090a8b64f991caa2fefaa9cbf47b7fb4ab6e3d4c" gracePeriod=25 Nov 03 20:34:24.199311 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.199279 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl" podUID="7b781aac-9f06-47eb-ac6a-16ba694919b2" containerName="payload-processing" containerID="cri-o://48a3ca9eac738b94dca0445bcd49a9b45d2dd43b7a0b2910c1c12ec3da3d7ebb" gracePeriod=25 Nov 03 20:34:24.469700 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.469668 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs" Nov 03 20:34:24.523052 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.523016 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef65466e-4f77-454d-9089-a7b0c636b563-kube-api-access-qqfr9\" (UniqueName: \"kubernetes.io/projected/ef65466e-4f77-454d-9089-a7b0c636b563-kube-api-access-qqfr9\") pod \"ef65466e-4f77-454d-9089-a7b0c636b563\" (UID: \"ef65466e-4f77-454d-9089-a7b0c636b563\") " Nov 03 20:34:24.531195 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.524240 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef65466e-4f77-454d-9089-a7b0c636b563-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef65466e-4f77-454d-9089-a7b0c636b563-plugins-config-volume\") pod \"ef65466e-4f77-454d-9089-a7b0c636b563\" (UID: \"ef65466e-4f77-454d-9089-a7b0c636b563\") " Nov 03 20:34:24.531195 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.524635 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef65466e-4f77-454d-9089-a7b0c636b563-plugins-config-volume" pod "ef65466e-4f77-454d-9089-a7b0c636b563" (UID: "ef65466e-4f77-454d-9089-a7b0c636b563"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:24.531195 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.525111 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef65466e-4f77-454d-9089-a7b0c636b563-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:24.535972 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.535318 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef65466e-4f77-454d-9089-a7b0c636b563-kube-api-access-qqfr9" pod "ef65466e-4f77-454d-9089-a7b0c636b563" (UID: "ef65466e-4f77-454d-9089-a7b0c636b563"). InnerVolumeSpecName "kube-api-access-qqfr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:24.582198 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.571344 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl" Nov 03 20:34:24.626607 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.626567 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b781aac-9f06-47eb-ac6a-16ba694919b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b781aac-9f06-47eb-ac6a-16ba694919b2-plugins-config-volume\") pod \"7b781aac-9f06-47eb-ac6a-16ba694919b2\" (UID: \"7b781aac-9f06-47eb-ac6a-16ba694919b2\") " Nov 03 20:34:24.626768 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.626617 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b781aac-9f06-47eb-ac6a-16ba694919b2-kube-api-access-v28zv\" (UniqueName: \"kubernetes.io/projected/7b781aac-9f06-47eb-ac6a-16ba694919b2-kube-api-access-v28zv\") pod \"7b781aac-9f06-47eb-ac6a-16ba694919b2\" (UID: \"7b781aac-9f06-47eb-ac6a-16ba694919b2\") " Nov 03 20:34:24.627056 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.627025 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b781aac-9f06-47eb-ac6a-16ba694919b2-plugins-config-volume" pod "7b781aac-9f06-47eb-ac6a-16ba694919b2" (UID: "7b781aac-9f06-47eb-ac6a-16ba694919b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:24.627327 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.627299 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b781aac-9f06-47eb-ac6a-16ba694919b2-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:24.627413 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.627327 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqfr9\" (UniqueName: \"kubernetes.io/projected/ef65466e-4f77-454d-9089-a7b0c636b563-kube-api-access-qqfr9\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:24.629309 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.629281 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b781aac-9f06-47eb-ac6a-16ba694919b2-kube-api-access-v28zv" pod "7b781aac-9f06-47eb-ac6a-16ba694919b2" (UID: "7b781aac-9f06-47eb-ac6a-16ba694919b2"). InnerVolumeSpecName "kube-api-access-v28zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:24.728281 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.728209 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v28zv\" (UniqueName: \"kubernetes.io/projected/7b781aac-9f06-47eb-ac6a-16ba694919b2-kube-api-access-v28zv\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:24.876276 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.876242 4918 generic.go:356] "Generic (PLEG): container finished" podID="ef65466e-4f77-454d-9089-a7b0c636b563" containerID="da6a647af7b41ab083ffd5d3090a8b64f991caa2fefaa9cbf47b7fb4ab6e3d4c" exitCode=0 Nov 03 20:34:24.876745 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.876289 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs" Nov 03 20:34:24.876745 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.876349 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs" event={"ID":"ef65466e-4f77-454d-9089-a7b0c636b563","Type":"ContainerDied","Data":"da6a647af7b41ab083ffd5d3090a8b64f991caa2fefaa9cbf47b7fb4ab6e3d4c"} Nov 03 20:34:24.876745 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.876442 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs" event={"ID":"ef65466e-4f77-454d-9089-a7b0c636b563","Type":"ContainerDied","Data":"05a97d6bef5f13a00d40686a5fd1e17c4393b5257b14ec696444f2c74d417741"} Nov 03 20:34:24.876745 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.876488 4918 scope.go:122] "RemoveContainer" containerID="da6a647af7b41ab083ffd5d3090a8b64f991caa2fefaa9cbf47b7fb4ab6e3d4c" Nov 03 20:34:24.877910 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.877890 4918 generic.go:356] "Generic (PLEG): container finished" podID="7b781aac-9f06-47eb-ac6a-16ba694919b2" containerID="48a3ca9eac738b94dca0445bcd49a9b45d2dd43b7a0b2910c1c12ec3da3d7ebb" exitCode=0 Nov 03 20:34:24.878111 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.878078 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl" event={"ID":"7b781aac-9f06-47eb-ac6a-16ba694919b2","Type":"ContainerDied","Data":"48a3ca9eac738b94dca0445bcd49a9b45d2dd43b7a0b2910c1c12ec3da3d7ebb"} Nov 03 20:34:24.878215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.878118 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl" event={"ID":"7b781aac-9f06-47eb-ac6a-16ba694919b2","Type":"ContainerDied","Data":"2ab727394bbe30a8ee865c54f50127ff6c8bcec36587d259901057bab3e882cc"} Nov 03 20:34:24.878273 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.878264 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl" Nov 03 20:34:24.889430 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.889412 4918 scope.go:122] "RemoveContainer" containerID="da6a647af7b41ab083ffd5d3090a8b64f991caa2fefaa9cbf47b7fb4ab6e3d4c" Nov 03 20:34:24.889774 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:34:24.889743 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da6a647af7b41ab083ffd5d3090a8b64f991caa2fefaa9cbf47b7fb4ab6e3d4c\": container with ID starting with da6a647af7b41ab083ffd5d3090a8b64f991caa2fefaa9cbf47b7fb4ab6e3d4c not found: ID does not exist" containerID="da6a647af7b41ab083ffd5d3090a8b64f991caa2fefaa9cbf47b7fb4ab6e3d4c" Nov 03 20:34:24.889852 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.889782 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da6a647af7b41ab083ffd5d3090a8b64f991caa2fefaa9cbf47b7fb4ab6e3d4c"} err="failed to get container status \"da6a647af7b41ab083ffd5d3090a8b64f991caa2fefaa9cbf47b7fb4ab6e3d4c\": rpc error: code = NotFound desc = could not find container \"da6a647af7b41ab083ffd5d3090a8b64f991caa2fefaa9cbf47b7fb4ab6e3d4c\": container with ID starting with da6a647af7b41ab083ffd5d3090a8b64f991caa2fefaa9cbf47b7fb4ab6e3d4c not found: ID does not exist" Nov 03 20:34:24.889852 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.889806 4918 scope.go:122] "RemoveContainer" containerID="48a3ca9eac738b94dca0445bcd49a9b45d2dd43b7a0b2910c1c12ec3da3d7ebb" Nov 03 20:34:24.904357 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.904334 4918 scope.go:122] "RemoveContainer" containerID="48a3ca9eac738b94dca0445bcd49a9b45d2dd43b7a0b2910c1c12ec3da3d7ebb" Nov 03 20:34:24.905370 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:34:24.905340 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a3ca9eac738b94dca0445bcd49a9b45d2dd43b7a0b2910c1c12ec3da3d7ebb\": container with ID starting with 48a3ca9eac738b94dca0445bcd49a9b45d2dd43b7a0b2910c1c12ec3da3d7ebb not found: ID does not exist" containerID="48a3ca9eac738b94dca0445bcd49a9b45d2dd43b7a0b2910c1c12ec3da3d7ebb" Nov 03 20:34:24.905481 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.905379 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a3ca9eac738b94dca0445bcd49a9b45d2dd43b7a0b2910c1c12ec3da3d7ebb"} err="failed to get container status \"48a3ca9eac738b94dca0445bcd49a9b45d2dd43b7a0b2910c1c12ec3da3d7ebb\": rpc error: code = NotFound desc = could not find container \"48a3ca9eac738b94dca0445bcd49a9b45d2dd43b7a0b2910c1c12ec3da3d7ebb\": container with ID starting with 48a3ca9eac738b94dca0445bcd49a9b45d2dd43b7a0b2910c1c12ec3da3d7ebb not found: ID does not exist" Nov 03 20:34:24.906671 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.906645 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs"] Nov 03 20:34:24.912113 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.912087 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ee4c78-75b68c4f8c-rv5bs"] Nov 03 20:34:24.926652 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.926622 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl"] Nov 03 20:34:24.933438 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:24.933415 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl"] Nov 03 20:34:25.215734 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:25.215698 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b781aac-9f06-47eb-ac6a-16ba694919b2" path="/var/lib/kubelet/pods/7b781aac-9f06-47eb-ac6a-16ba694919b2/volumes" Nov 03 20:34:25.216138 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:25.216125 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7a163ac-b7b9-460f-a0e5-73faac7e3f48" path="/var/lib/kubelet/pods/c7a163ac-b7b9-460f-a0e5-73faac7e3f48/volumes" Nov 03 20:34:25.216540 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:25.216525 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef65466e-4f77-454d-9089-a7b0c636b563" path="/var/lib/kubelet/pods/ef65466e-4f77-454d-9089-a7b0c636b563/volumes" Nov 03 20:34:25.476457 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:25.476350 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ee4c78-54d9d66478-z75nl" podUID="7b781aac-9f06-47eb-ac6a-16ba694919b2" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.101:50052\" within 1s: context deadline exceeded" Nov 03 20:34:25.476457 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:34:25.476430 4918 logging.go:55] [core] [Channel #5291 SubChannel #5292]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.101:50052", ServerName: "10.132.0.101:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.101:50052: operation was canceled" Nov 03 20:34:28.208484 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:34:28.208439 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:34:30.725772 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:30.725731 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5"] Nov 03 20:34:30.726276 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:30.726012 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" podUID="af357454-5d01-46c9-a1ef-bf09169443ee" containerName="istio-proxy" containerID="cri-o://adb53663e85dc17295b9550847c3c7721a0e6c4c8ee16cd2bc7da740973482c9" gracePeriod=30 Nov 03 20:34:33.750732 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:33.750691 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz"] Nov 03 20:34:33.751298 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:33.750977 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" podUID="5120ad5e-81f7-43c1-9e1d-f0407339bb0c" containerName="istio-proxy" containerID="cri-o://1aba7bd51ef0926c55f9b694d24bb37be6e60f8f1a23ee86900ff18f22fe2369" gracePeriod=30 Nov 03 20:34:34.482506 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.482469 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4"] Nov 03 20:34:34.483453 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483426 4918 cpu_manager.go:414] "containerMap: removing container" podUID="1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" containerName="istio-proxy" Nov 03 20:34:34.483517 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483457 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" containerName="istio-proxy" Nov 03 20:34:34.483517 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483479 4918 cpu_manager.go:414] "containerMap: removing container" podUID="81a01d22-8738-4955-9dc5-73d769266ddc" containerName="cleanup" Nov 03 20:34:34.483517 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483488 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a01d22-8738-4955-9dc5-73d769266ddc" containerName="cleanup" Nov 03 20:34:34.483517 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483503 4918 cpu_manager.go:414] "containerMap: removing container" podUID="11e6ea59-b2c6-409d-b2e8-4a4eec65938d" containerName="istio-proxy" Nov 03 20:34:34.483517 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483513 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e6ea59-b2c6-409d-b2e8-4a4eec65938d" containerName="istio-proxy" Nov 03 20:34:34.483679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483525 4918 cpu_manager.go:414] "containerMap: removing container" podUID="6dad3fad-396e-4647-a9bf-a7a906fdd1fe" containerName="istio-proxy" Nov 03 20:34:34.483679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483532 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dad3fad-396e-4647-a9bf-a7a906fdd1fe" containerName="istio-proxy" Nov 03 20:34:34.483679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483541 4918 cpu_manager.go:414] "containerMap: removing container" podUID="8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0" containerName="main" Nov 03 20:34:34.483679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483549 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0" containerName="main" Nov 03 20:34:34.483679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483567 4918 cpu_manager.go:414] "containerMap: removing container" podUID="a96cad21-e85e-4757-9e72-6b350738e230" containerName="payload-processing" Nov 03 20:34:34.483679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483577 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a96cad21-e85e-4757-9e72-6b350738e230" containerName="payload-processing" Nov 03 20:34:34.483679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483589 4918 cpu_manager.go:414] "containerMap: removing container" podUID="c4cccb65-6424-4e0b-85bf-6f1fba8d5572" containerName="cleanup" Nov 03 20:34:34.483679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483596 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4cccb65-6424-4e0b-85bf-6f1fba8d5572" containerName="cleanup" Nov 03 20:34:34.483679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483609 4918 cpu_manager.go:414] "containerMap: removing container" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" Nov 03 20:34:34.483679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483618 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" Nov 03 20:34:34.483679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483629 4918 cpu_manager.go:414] "containerMap: removing container" podUID="81a01d22-8738-4955-9dc5-73d769266ddc" containerName="cleanup" Nov 03 20:34:34.483679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483637 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a01d22-8738-4955-9dc5-73d769266ddc" containerName="cleanup" Nov 03 20:34:34.483679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483652 4918 cpu_manager.go:414] "containerMap: removing container" podUID="4c77123e-9c94-4299-a7e0-1759315d24fd" containerName="maas-api" Nov 03 20:34:34.483679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483661 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c77123e-9c94-4299-a7e0-1759315d24fd" containerName="maas-api" Nov 03 20:34:34.483679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483679 4918 cpu_manager.go:414] "containerMap: removing container" podUID="eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" containerName="istio-proxy" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483688 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" containerName="istio-proxy" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483699 4918 cpu_manager.go:414] "containerMap: removing container" podUID="b2461bee-954b-481d-a740-379482984a4a" containerName="istio-proxy" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483706 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2461bee-954b-481d-a740-379482984a4a" containerName="istio-proxy" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483718 4918 cpu_manager.go:414] "containerMap: removing container" podUID="b6290d9f-ddd4-40d2-809b-110249e01125" containerName="istio-proxy" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483725 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6290d9f-ddd4-40d2-809b-110249e01125" containerName="istio-proxy" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483738 4918 cpu_manager.go:414] "containerMap: removing container" podUID="cc721d0a-faab-45c5-8476-9aec206a65c3" containerName="cleanup" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483744 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc721d0a-faab-45c5-8476-9aec206a65c3" containerName="cleanup" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483752 4918 cpu_manager.go:414] "containerMap: removing container" podUID="2228ed1d-945a-4b33-ba98-ee8b52cbf902" containerName="cleanup" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483759 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2228ed1d-945a-4b33-ba98-ee8b52cbf902" containerName="cleanup" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483767 4918 cpu_manager.go:414] "containerMap: removing container" podUID="74173946-29b1-4686-985c-73d7f43b51bf" containerName="maas-api" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483774 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74173946-29b1-4686-985c-73d7f43b51bf" containerName="maas-api" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483786 4918 cpu_manager.go:414] "containerMap: removing container" podUID="f6f6c399-4702-41b4-b4f4-8920d34d7cfb" containerName="istio-proxy" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483792 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f6c399-4702-41b4-b4f4-8920d34d7cfb" containerName="istio-proxy" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483804 4918 cpu_manager.go:414] "containerMap: removing container" podUID="1a6a964b-7ff6-4c06-b46f-499c03cfd744" containerName="payload-processing" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483811 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a6a964b-7ff6-4c06-b46f-499c03cfd744" containerName="payload-processing" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483820 4918 cpu_manager.go:414] "containerMap: removing container" podUID="67a9be3d-a3d7-450b-926f-96fa79503381" containerName="revoke-keys" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483827 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67a9be3d-a3d7-450b-926f-96fa79503381" containerName="revoke-keys" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483838 4918 cpu_manager.go:414] "containerMap: removing container" podUID="46b614fc-7245-4556-886c-3f9118db35ad" containerName="istio-proxy" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483845 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b614fc-7245-4556-886c-3f9118db35ad" containerName="istio-proxy" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483853 4918 cpu_manager.go:414] "containerMap: removing container" podUID="ea246681-cf7a-40b9-97d1-9c1d2594c3ec" containerName="istio-proxy" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483860 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea246681-cf7a-40b9-97d1-9c1d2594c3ec" containerName="istio-proxy" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483870 4918 cpu_manager.go:414] "containerMap: removing container" podUID="4e6db5de-2704-45a9-8e69-a78d3cf1fb5a" containerName="payload-pre-processing" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483877 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e6db5de-2704-45a9-8e69-a78d3cf1fb5a" containerName="payload-pre-processing" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483889 4918 cpu_manager.go:414] "containerMap: removing container" podUID="a842122e-88c2-4a73-aabf-727008e66cd0" containerName="main" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483896 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a842122e-88c2-4a73-aabf-727008e66cd0" containerName="main" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483913 4918 cpu_manager.go:414] "containerMap: removing container" podUID="c7a163ac-b7b9-460f-a0e5-73faac7e3f48" containerName="maas-api" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483920 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a163ac-b7b9-460f-a0e5-73faac7e3f48" containerName="maas-api" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483931 4918 cpu_manager.go:414] "containerMap: removing container" podUID="0832a6d3-76cd-4eb6-9879-c880f98a786f" containerName="cleanup" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483938 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0832a6d3-76cd-4eb6-9879-c880f98a786f" containerName="cleanup" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483948 4918 cpu_manager.go:414] "containerMap: removing container" podUID="7a2653fa-4e55-48c8-a756-1f24c4c8412a" containerName="cleanup" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483956 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2653fa-4e55-48c8-a756-1f24c4c8412a" containerName="cleanup" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483975 4918 cpu_manager.go:414] "containerMap: removing container" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483982 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483991 4918 cpu_manager.go:414] "containerMap: removing container" podUID="97238908-155f-4656-8767-fae2a876d653" containerName="main" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.483998 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97238908-155f-4656-8767-fae2a876d653" containerName="main" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484007 4918 cpu_manager.go:414] "containerMap: removing container" podUID="c4cccb65-6424-4e0b-85bf-6f1fba8d5572" containerName="cleanup" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484013 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4cccb65-6424-4e0b-85bf-6f1fba8d5572" containerName="cleanup" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484023 4918 cpu_manager.go:414] "containerMap: removing container" podUID="9fb1ff44-4b90-47da-a252-d5771f160330" containerName="istio-proxy" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484030 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb1ff44-4b90-47da-a252-d5771f160330" containerName="istio-proxy" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484042 4918 cpu_manager.go:414] "containerMap: removing container" podUID="2f8f2527-d745-4a36-b656-594d782b0db3" containerName="istio-proxy" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484050 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8f2527-d745-4a36-b656-594d782b0db3" containerName="istio-proxy" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484060 4918 cpu_manager.go:414] "containerMap: removing container" podUID="b1203aec-cde6-46b4-acf9-991475721fd7" containerName="main" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484067 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1203aec-cde6-46b4-acf9-991475721fd7" containerName="main" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484077 4918 cpu_manager.go:414] "containerMap: removing container" podUID="752114fb-ce1f-4ddb-8a69-75c8a44a364f" containerName="cleanup" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484084 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752114fb-ce1f-4ddb-8a69-75c8a44a364f" containerName="cleanup" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484098 4918 cpu_manager.go:414] "containerMap: removing container" podUID="14e66787-45ea-4e62-83f8-e14a0388b915" containerName="maas-api" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484105 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14e66787-45ea-4e62-83f8-e14a0388b915" containerName="maas-api" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484114 4918 cpu_manager.go:414] "containerMap: removing container" podUID="cc721d0a-faab-45c5-8476-9aec206a65c3" containerName="cleanup" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484121 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc721d0a-faab-45c5-8476-9aec206a65c3" containerName="cleanup" Nov 03 20:34:34.484120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484137 4918 cpu_manager.go:414] "containerMap: removing container" podUID="c1759364-4aca-4b1d-9f83-a81bc5829924" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484144 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1759364-4aca-4b1d-9f83-a81bc5829924" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484160 4918 cpu_manager.go:414] "containerMap: removing container" podUID="a7ea5a18-974d-456d-88b5-e05b8cd33f9c" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484167 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ea5a18-974d-456d-88b5-e05b8cd33f9c" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484195 4918 cpu_manager.go:414] "containerMap: removing container" podUID="3b00acc2-340d-4fec-bab6-efadbf64e74d" containerName="payload-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484203 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b00acc2-340d-4fec-bab6-efadbf64e74d" containerName="payload-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484217 4918 cpu_manager.go:414] "containerMap: removing container" podUID="7b781aac-9f06-47eb-ac6a-16ba694919b2" containerName="payload-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484224 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b781aac-9f06-47eb-ac6a-16ba694919b2" containerName="payload-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484234 4918 cpu_manager.go:414] "containerMap: removing container" podUID="7103c18e-609f-41ed-862a-13a571202b51" containerName="payload-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484241 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7103c18e-609f-41ed-862a-13a571202b51" containerName="payload-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484251 4918 cpu_manager.go:414] "containerMap: removing container" podUID="3ded5532-c475-49a9-8e49-d4f547837159" containerName="payload-pre-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484257 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ded5532-c475-49a9-8e49-d4f547837159" containerName="payload-pre-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484277 4918 cpu_manager.go:414] "containerMap: removing container" podUID="ef65466e-4f77-454d-9089-a7b0c636b563" containerName="payload-pre-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484284 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef65466e-4f77-454d-9089-a7b0c636b563" containerName="payload-pre-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484295 4918 cpu_manager.go:414] "containerMap: removing container" podUID="9fb1ff44-4b90-47da-a252-d5771f160330" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484301 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb1ff44-4b90-47da-a252-d5771f160330" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484314 4918 cpu_manager.go:414] "containerMap: removing container" podUID="769e99e9-f468-4747-ac31-5fe11e893dce" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484321 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769e99e9-f468-4747-ac31-5fe11e893dce" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484333 4918 cpu_manager.go:414] "containerMap: removing container" podUID="e925174a-652a-4443-bf13-7b979933a297" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484340 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e925174a-652a-4443-bf13-7b979933a297" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484353 4918 cpu_manager.go:414] "containerMap: removing container" podUID="0832a6d3-76cd-4eb6-9879-c880f98a786f" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484360 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0832a6d3-76cd-4eb6-9879-c880f98a786f" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484371 4918 cpu_manager.go:414] "containerMap: removing container" podUID="2f8f2527-d745-4a36-b656-594d782b0db3" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484378 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8f2527-d745-4a36-b656-594d782b0db3" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484388 4918 cpu_manager.go:414] "containerMap: removing container" podUID="0832a6d3-76cd-4eb6-9879-c880f98a786f" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484395 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0832a6d3-76cd-4eb6-9879-c880f98a786f" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484405 4918 cpu_manager.go:414] "containerMap: removing container" podUID="df7b9bb4-44fb-4272-a6c7-90e71d051f46" containerName="main" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484412 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df7b9bb4-44fb-4272-a6c7-90e71d051f46" containerName="main" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484421 4918 cpu_manager.go:414] "containerMap: removing container" podUID="b612e8de-dd1e-47db-a7f8-28eb9e4149f9" containerName="payload-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484429 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b612e8de-dd1e-47db-a7f8-28eb9e4149f9" containerName="payload-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484443 4918 cpu_manager.go:414] "containerMap: removing container" podUID="2ea5be43-cacb-48ac-89bb-ca19bdc86522" containerName="payload-pre-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484450 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea5be43-cacb-48ac-89bb-ca19bdc86522" containerName="payload-pre-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484464 4918 cpu_manager.go:414] "containerMap: removing container" podUID="7a2653fa-4e55-48c8-a756-1f24c4c8412a" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484473 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2653fa-4e55-48c8-a756-1f24c4c8412a" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484485 4918 cpu_manager.go:414] "containerMap: removing container" podUID="4b47d59c-60aa-483d-82bd-35a5af6b10b6" containerName="maas-api" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484492 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b47d59c-60aa-483d-82bd-35a5af6b10b6" containerName="maas-api" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484502 4918 cpu_manager.go:414] "containerMap: removing container" podUID="752114fb-ce1f-4ddb-8a69-75c8a44a364f" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484509 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752114fb-ce1f-4ddb-8a69-75c8a44a364f" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484520 4918 cpu_manager.go:414] "containerMap: removing container" podUID="d986da59-41d5-4c2e-8832-e7e79e5e2b19" containerName="maas-api" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484527 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d986da59-41d5-4c2e-8832-e7e79e5e2b19" containerName="maas-api" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484544 4918 cpu_manager.go:414] "containerMap: removing container" podUID="752114fb-ce1f-4ddb-8a69-75c8a44a364f" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484552 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752114fb-ce1f-4ddb-8a69-75c8a44a364f" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484564 4918 cpu_manager.go:414] "containerMap: removing container" podUID="d270ff2c-258f-4c66-b50e-1bbb2494816d" containerName="payload-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484572 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d270ff2c-258f-4c66-b50e-1bbb2494816d" containerName="payload-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484583 4918 cpu_manager.go:414] "containerMap: removing container" podUID="2228ed1d-945a-4b33-ba98-ee8b52cbf902" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484591 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2228ed1d-945a-4b33-ba98-ee8b52cbf902" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484606 4918 cpu_manager.go:414] "containerMap: removing container" podUID="cc721d0a-faab-45c5-8476-9aec206a65c3" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484613 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc721d0a-faab-45c5-8476-9aec206a65c3" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484773 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1203aec-cde6-46b4-acf9-991475721fd7" containerName="main" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484786 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="752114fb-ce1f-4ddb-8a69-75c8a44a364f" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484799 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="14e66787-45ea-4e62-83f8-e14a0388b915" containerName="maas-api" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484809 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc721d0a-faab-45c5-8476-9aec206a65c3" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484818 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1759364-4aca-4b1d-9f83-a81bc5829924" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484832 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ea5a18-974d-456d-88b5-e05b8cd33f9c" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484841 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484851 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b00acc2-340d-4fec-bab6-efadbf64e74d" containerName="payload-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484863 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b781aac-9f06-47eb-ac6a-16ba694919b2" containerName="payload-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484871 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7103c18e-609f-41ed-862a-13a571202b51" containerName="payload-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484882 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ded5532-c475-49a9-8e49-d4f547837159" containerName="payload-pre-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484894 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef65466e-4f77-454d-9089-a7b0c636b563" containerName="payload-pre-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484903 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb1ff44-4b90-47da-a252-d5771f160330" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484913 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="769e99e9-f468-4747-ac31-5fe11e893dce" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484924 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e925174a-652a-4443-bf13-7b979933a297" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484932 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0832a6d3-76cd-4eb6-9879-c880f98a786f" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484941 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8f2527-d745-4a36-b656-594d782b0db3" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484952 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0832a6d3-76cd-4eb6-9879-c880f98a786f" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484960 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="df7b9bb4-44fb-4272-a6c7-90e71d051f46" containerName="main" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484968 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b612e8de-dd1e-47db-a7f8-28eb9e4149f9" containerName="payload-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484978 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea5be43-cacb-48ac-89bb-ca19bdc86522" containerName="payload-pre-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484988 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2653fa-4e55-48c8-a756-1f24c4c8412a" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.484995 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2653fa-4e55-48c8-a756-1f24c4c8412a" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485005 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b47d59c-60aa-483d-82bd-35a5af6b10b6" containerName="maas-api" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485014 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="752114fb-ce1f-4ddb-8a69-75c8a44a364f" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485025 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d986da59-41d5-4c2e-8832-e7e79e5e2b19" containerName="maas-api" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485034 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="752114fb-ce1f-4ddb-8a69-75c8a44a364f" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485044 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d270ff2c-258f-4c66-b50e-1bbb2494816d" containerName="payload-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485054 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4cccb65-6424-4e0b-85bf-6f1fba8d5572" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485063 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2228ed1d-945a-4b33-ba98-ee8b52cbf902" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485073 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a01d22-8738-4955-9dc5-73d769266ddc" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485082 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e6ea59-b2c6-409d-b2e8-4a4eec65938d" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485090 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dad3fad-396e-4647-a9bf-a7a906fdd1fe" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485099 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="8de9e587-15c4-48b3-bcfe-ac1e7a2b0fa0" containerName="main" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485107 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a96cad21-e85e-4757-9e72-6b350738e230" containerName="payload-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485118 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4cccb65-6424-4e0b-85bf-6f1fba8d5572" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485127 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b14fd53e-9193-44a0-9693-446fe067a841" containerName="main" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485137 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a01d22-8738-4955-9dc5-73d769266ddc" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485145 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c77123e-9c94-4299-a7e0-1759315d24fd" containerName="maas-api" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485153 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeae65a9-5d82-4c1c-ade9-2c0cbb210dc5" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485164 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2461bee-954b-481d-a740-379482984a4a" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485190 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6290d9f-ddd4-40d2-809b-110249e01125" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485202 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc721d0a-faab-45c5-8476-9aec206a65c3" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485211 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2228ed1d-945a-4b33-ba98-ee8b52cbf902" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485218 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="74173946-29b1-4686-985c-73d7f43b51bf" containerName="maas-api" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485227 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f6c399-4702-41b4-b4f4-8920d34d7cfb" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485236 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a6a964b-7ff6-4c06-b46f-499c03cfd744" containerName="payload-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485246 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="67a9be3d-a3d7-450b-926f-96fa79503381" containerName="revoke-keys" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485255 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b614fc-7245-4556-886c-3f9118db35ad" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485269 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea246681-cf7a-40b9-97d1-9c1d2594c3ec" containerName="istio-proxy" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485282 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e6db5de-2704-45a9-8e69-a78d3cf1fb5a" containerName="payload-pre-processing" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485296 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a842122e-88c2-4a73-aabf-727008e66cd0" containerName="main" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485314 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a163ac-b7b9-460f-a0e5-73faac7e3f48" containerName="maas-api" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485323 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2653fa-4e55-48c8-a756-1f24c4c8412a" containerName="cleanup" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485335 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5ce248-0c9e-4f39-9130-1b2aad5beb65" containerName="main" Nov 03 20:34:34.485636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.485347 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="97238908-155f-4656-8767-fae2a876d653" containerName="main" Nov 03 20:34:34.491190 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.491154 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.496909 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.496884 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c60d2a-openshift-default-dockercfg-c4xvf\"" Nov 03 20:34:34.499662 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.499633 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4"] Nov 03 20:34:34.560900 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.560866 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-istio-data\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.561069 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.560911 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-istio-envoy\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.561069 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.560983 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942fc70c-abe0-4988-b6f9-c9babff54817-istio-podinfo\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.561152 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.561064 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-credential-socket\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.561152 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.561100 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942fc70c-abe0-4988-b6f9-c9babff54817-istio-token\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.561152 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.561149 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942fc70c-abe0-4988-b6f9-c9babff54817-istiod-ca-cert\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.561295 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.561204 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-workload-socket\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.561327 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.561300 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7zxn\" (UniqueName: \"kubernetes.io/projected/942fc70c-abe0-4988-b6f9-c9babff54817-kube-api-access-n7zxn\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.561380 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.561367 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-workload-certs\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.662740 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.662703 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-istio-data\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.662740 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.662752 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-istio-envoy\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.662997 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.662861 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942fc70c-abe0-4988-b6f9-c9babff54817-istio-podinfo\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.662997 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.662893 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-credential-socket\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.662997 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.662914 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942fc70c-abe0-4988-b6f9-c9babff54817-istio-token\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.662997 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.662982 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942fc70c-abe0-4988-b6f9-c9babff54817-istiod-ca-cert\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.663277 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.663014 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-workload-socket\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.663277 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.663140 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7zxn\" (UniqueName: \"kubernetes.io/projected/942fc70c-abe0-4988-b6f9-c9babff54817-kube-api-access-n7zxn\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.663277 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.663252 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-workload-certs\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.664469 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.663773 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-istio-data\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.664469 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.663880 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-credential-socket\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.664469 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.663997 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-workload-socket\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.664469 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.664082 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-workload-certs\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.664469 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.664421 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942fc70c-abe0-4988-b6f9-c9babff54817-istiod-ca-cert\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.665790 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.665761 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-istio-envoy\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.666012 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.665973 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942fc70c-abe0-4988-b6f9-c9babff54817-istio-podinfo\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.684688 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.684663 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942fc70c-abe0-4988-b6f9-c9babff54817-istio-token\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.685146 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.685117 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7zxn\" (UniqueName: \"kubernetes.io/projected/942fc70c-abe0-4988-b6f9-c9babff54817-kube-api-access-n7zxn\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:34.818718 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:34.818684 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:35.108498 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:35.108466 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4"] Nov 03 20:34:35.110374 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:34:35.110346 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942fc70c_abe0_4988_b6f9_c9babff54817.slice/crio-b3dc8323fed4d52e2da8b073e54f036533821b3d5ed4bf8c69f1b7aa2017cc9e WatchSource:0}: Error finding container b3dc8323fed4d52e2da8b073e54f036533821b3d5ed4bf8c69f1b7aa2017cc9e: Status 404 returned error can't find the container with id b3dc8323fed4d52e2da8b073e54f036533821b3d5ed4bf8c69f1b7aa2017cc9e Nov 03 20:34:35.113040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:35.113001 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:34:35.113160 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:35.113067 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:34:35.113160 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:35.113100 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:34:36.008522 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.008433 4918 generic.go:356] "Generic (PLEG): container finished" podID="af357454-5d01-46c9-a1ef-bf09169443ee" containerID="adb53663e85dc17295b9550847c3c7721a0e6c4c8ee16cd2bc7da740973482c9" exitCode=0 Nov 03 20:34:36.009209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.009162 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" event={"ID":"af357454-5d01-46c9-a1ef-bf09169443ee","Type":"ContainerDied","Data":"adb53663e85dc17295b9550847c3c7721a0e6c4c8ee16cd2bc7da740973482c9"} Nov 03 20:34:36.009442 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.009424 4918 scope.go:122] "RemoveContainer" containerID="20d4b76c2686d47d81d337cd1b4d9edd311a9e9e0c4da64bca25df990eea9a50" Nov 03 20:34:36.014096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.014068 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" event={"ID":"942fc70c-abe0-4988-b6f9-c9babff54817","Type":"ContainerStarted","Data":"b040b847214a2009d427e3ea93b4bd146323adf4602fb0033ea9aab279e6e353"} Nov 03 20:34:36.014346 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.014328 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" event={"ID":"942fc70c-abe0-4988-b6f9-c9babff54817","Type":"ContainerStarted","Data":"b3dc8323fed4d52e2da8b073e54f036533821b3d5ed4bf8c69f1b7aa2017cc9e"} Nov 03 20:34:36.050378 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.050320 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" podStartSLOduration=2.050299941 podStartE2EDuration="2.050299941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:36.0487249 +0000 UTC m=+2258.980592446" watchObservedRunningTime="2026-11-03 20:34:36.050299941 +0000 UTC m=+2258.982167453" Nov 03 20:34:36.115303 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.115272 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:34:36.202859 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.202820 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-workload-certs\") pod \"af357454-5d01-46c9-a1ef-bf09169443ee\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " Nov 03 20:34:36.203072 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.202874 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-credential-socket\") pod \"af357454-5d01-46c9-a1ef-bf09169443ee\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " Nov 03 20:34:36.203072 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.202956 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af357454-5d01-46c9-a1ef-bf09169443ee-kube-api-access-qhw7q\" (UniqueName: \"kubernetes.io/projected/af357454-5d01-46c9-a1ef-bf09169443ee-kube-api-access-qhw7q\") pod \"af357454-5d01-46c9-a1ef-bf09169443ee\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " Nov 03 20:34:36.203072 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.202984 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-istio-data\") pod \"af357454-5d01-46c9-a1ef-bf09169443ee\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " Nov 03 20:34:36.203072 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.203011 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-workload-socket\") pod \"af357454-5d01-46c9-a1ef-bf09169443ee\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " Nov 03 20:34:36.203072 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.203044 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-istio-envoy\") pod \"af357454-5d01-46c9-a1ef-bf09169443ee\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " Nov 03 20:34:36.203377 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.203132 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af357454-5d01-46c9-a1ef-bf09169443ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af357454-5d01-46c9-a1ef-bf09169443ee-istio-podinfo\") pod \"af357454-5d01-46c9-a1ef-bf09169443ee\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " Nov 03 20:34:36.203377 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.203133 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-workload-certs" pod "af357454-5d01-46c9-a1ef-bf09169443ee" (UID: "af357454-5d01-46c9-a1ef-bf09169443ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:36.203377 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.203206 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-credential-socket" pod "af357454-5d01-46c9-a1ef-bf09169443ee" (UID: "af357454-5d01-46c9-a1ef-bf09169443ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:36.203377 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.203232 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af357454-5d01-46c9-a1ef-bf09169443ee-istio-token\" (UniqueName: \"kubernetes.io/projected/af357454-5d01-46c9-a1ef-bf09169443ee-istio-token\") pod \"af357454-5d01-46c9-a1ef-bf09169443ee\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " Nov 03 20:34:36.203377 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.203354 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-workload-socket" pod "af357454-5d01-46c9-a1ef-bf09169443ee" (UID: "af357454-5d01-46c9-a1ef-bf09169443ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:36.203377 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.203365 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-istio-data" pod "af357454-5d01-46c9-a1ef-bf09169443ee" (UID: "af357454-5d01-46c9-a1ef-bf09169443ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:36.203693 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.203361 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af357454-5d01-46c9-a1ef-bf09169443ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af357454-5d01-46c9-a1ef-bf09169443ee-istiod-ca-cert\") pod \"af357454-5d01-46c9-a1ef-bf09169443ee\" (UID: \"af357454-5d01-46c9-a1ef-bf09169443ee\") " Nov 03 20:34:36.203748 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.203704 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af357454-5d01-46c9-a1ef-bf09169443ee-istiod-ca-cert" pod "af357454-5d01-46c9-a1ef-bf09169443ee" (UID: "af357454-5d01-46c9-a1ef-bf09169443ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:36.203937 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.203885 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af357454-5d01-46c9-a1ef-bf09169443ee-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:36.203937 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.203910 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:36.203937 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.203924 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:36.203937 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.203937 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:36.204102 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.203947 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:36.205965 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.205941 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af357454-5d01-46c9-a1ef-bf09169443ee-istio-token" pod "af357454-5d01-46c9-a1ef-bf09169443ee" (UID: "af357454-5d01-46c9-a1ef-bf09169443ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:36.206074 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.206014 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af357454-5d01-46c9-a1ef-bf09169443ee-kube-api-access-qhw7q" pod "af357454-5d01-46c9-a1ef-bf09169443ee" (UID: "af357454-5d01-46c9-a1ef-bf09169443ee"). InnerVolumeSpecName "kube-api-access-qhw7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:36.206118 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.206079 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-istio-envoy" pod "af357454-5d01-46c9-a1ef-bf09169443ee" (UID: "af357454-5d01-46c9-a1ef-bf09169443ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:36.206256 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.206239 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af357454-5d01-46c9-a1ef-bf09169443ee-istio-podinfo" pod "af357454-5d01-46c9-a1ef-bf09169443ee" (UID: "af357454-5d01-46c9-a1ef-bf09169443ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:36.305548 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.305443 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af357454-5d01-46c9-a1ef-bf09169443ee-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:36.305548 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.305495 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhw7q\" (UniqueName: \"kubernetes.io/projected/af357454-5d01-46c9-a1ef-bf09169443ee-kube-api-access-qhw7q\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:36.305548 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.305515 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af357454-5d01-46c9-a1ef-bf09169443ee-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:36.305548 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.305530 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af357454-5d01-46c9-a1ef-bf09169443ee-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:36.834691 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.834652 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:36.862051 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.861662 4918 patch_prober.go:29] interesting pod/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 20:34:36.862051 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.861740 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" podUID="942fc70c-abe0-4988-b6f9-c9babff54817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 20:34:36.899343 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:36.899307 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66"] Nov 03 20:34:37.046658 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:37.046624 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" Nov 03 20:34:37.049343 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:37.048836 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5" event={"ID":"af357454-5d01-46c9-a1ef-bf09169443ee","Type":"ContainerDied","Data":"6b161c8c6845f95589681f45731142ba967abf86277a14f8db2e0e160b977b85"} Nov 03 20:34:37.049496 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:37.049381 4918 scope.go:122] "RemoveContainer" containerID="adb53663e85dc17295b9550847c3c7721a0e6c4c8ee16cd2bc7da740973482c9" Nov 03 20:34:37.081469 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:37.081439 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5"] Nov 03 20:34:37.086431 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:37.086379 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3e9176-openshift-default-695bff8877-zh6w5"] Nov 03 20:34:37.251202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:37.250433 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af357454-5d01-46c9-a1ef-bf09169443ee" path="/var/lib/kubelet/pods/af357454-5d01-46c9-a1ef-bf09169443ee/volumes" Nov 03 20:34:37.460412 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:37.460327 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc"] Nov 03 20:34:37.460611 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:37.460548 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" podUID="3e465e5f-cb99-475a-848a-a182ceccc56e" containerName="istio-proxy" containerID="cri-o://2dad895aa9ccdc55828c60f4e9f663fe0e96535c05dfd56982ea4b27fd257859" gracePeriod=30 Nov 03 20:34:37.826220 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:37.822626 4918 patch_prober.go:29] interesting pod/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 20:34:37.826220 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:37.822692 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" podUID="942fc70c-abe0-4988-b6f9-c9babff54817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 20:34:38.822201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:38.819561 4918 patch_prober.go:29] interesting pod/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 20:34:38.822201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:38.819621 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" podUID="942fc70c-abe0-4988-b6f9-c9babff54817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 20:34:39.063375 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.063215 4918 generic.go:356] "Generic (PLEG): container finished" podID="5120ad5e-81f7-43c1-9e1d-f0407339bb0c" containerID="1aba7bd51ef0926c55f9b694d24bb37be6e60f8f1a23ee86900ff18f22fe2369" exitCode=0 Nov 03 20:34:39.063375 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.063330 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" event={"ID":"5120ad5e-81f7-43c1-9e1d-f0407339bb0c","Type":"ContainerDied","Data":"1aba7bd51ef0926c55f9b694d24bb37be6e60f8f1a23ee86900ff18f22fe2369"} Nov 03 20:34:39.086695 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.086663 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:34:39.176090 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.176055 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-kube-api-access-hf9cn\" (UniqueName: \"kubernetes.io/projected/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-kube-api-access-hf9cn\") pod \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " Nov 03 20:34:39.176310 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.176101 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-data\") pod \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " Nov 03 20:34:39.176310 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.176160 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-token\" (UniqueName: \"kubernetes.io/projected/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-token\") pod \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " Nov 03 20:34:39.176310 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.176235 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-podinfo\") pod \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " Nov 03 20:34:39.176499 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.176323 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-workload-socket\") pod \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " Nov 03 20:34:39.176499 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.176367 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istiod-ca-cert\") pod \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " Nov 03 20:34:39.176499 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.176433 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-workload-certs\") pod \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " Nov 03 20:34:39.176655 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.176506 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-data" pod "5120ad5e-81f7-43c1-9e1d-f0407339bb0c" (UID: "5120ad5e-81f7-43c1-9e1d-f0407339bb0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:39.176655 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.176537 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-envoy\") pod \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " Nov 03 20:34:39.176655 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.176591 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-credential-socket\") pod \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\" (UID: \"5120ad5e-81f7-43c1-9e1d-f0407339bb0c\") " Nov 03 20:34:39.176839 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.176757 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-workload-certs" pod "5120ad5e-81f7-43c1-9e1d-f0407339bb0c" (UID: "5120ad5e-81f7-43c1-9e1d-f0407339bb0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:39.176945 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.176919 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istiod-ca-cert" pod "5120ad5e-81f7-43c1-9e1d-f0407339bb0c" (UID: "5120ad5e-81f7-43c1-9e1d-f0407339bb0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:39.177102 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.176965 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-credential-socket" pod "5120ad5e-81f7-43c1-9e1d-f0407339bb0c" (UID: "5120ad5e-81f7-43c1-9e1d-f0407339bb0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:39.177240 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.177093 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-workload-socket" pod "5120ad5e-81f7-43c1-9e1d-f0407339bb0c" (UID: "5120ad5e-81f7-43c1-9e1d-f0407339bb0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:39.177339 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.177323 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:39.177400 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.177355 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:39.177400 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.177373 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:39.179118 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.179084 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-podinfo" pod "5120ad5e-81f7-43c1-9e1d-f0407339bb0c" (UID: "5120ad5e-81f7-43c1-9e1d-f0407339bb0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:39.179249 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.179123 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-token" pod "5120ad5e-81f7-43c1-9e1d-f0407339bb0c" (UID: "5120ad5e-81f7-43c1-9e1d-f0407339bb0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:39.179417 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.179392 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-kube-api-access-hf9cn" pod "5120ad5e-81f7-43c1-9e1d-f0407339bb0c" (UID: "5120ad5e-81f7-43c1-9e1d-f0407339bb0c"). InnerVolumeSpecName "kube-api-access-hf9cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:39.180320 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.180297 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-envoy" pod "5120ad5e-81f7-43c1-9e1d-f0407339bb0c" (UID: "5120ad5e-81f7-43c1-9e1d-f0407339bb0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:39.278634 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.278601 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:39.278634 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.278634 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf9cn\" (UniqueName: \"kubernetes.io/projected/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-kube-api-access-hf9cn\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:39.278840 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.278646 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:39.278840 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.278655 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:39.278840 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.278664 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:39.278840 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.278673 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5120ad5e-81f7-43c1-9e1d-f0407339bb0c-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:39.821211 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.820604 4918 patch_prober.go:29] interesting pod/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 20:34:39.821211 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:39.820666 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" podUID="942fc70c-abe0-4988-b6f9-c9babff54817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 20:34:40.070680 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:40.070644 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" Nov 03 20:34:40.071208 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:40.070676 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz" event={"ID":"5120ad5e-81f7-43c1-9e1d-f0407339bb0c","Type":"ContainerDied","Data":"fffeb42847a93d4686cc5cfe38a8a5009d4291d8ac343a2d3a794ac1ff19ff59"} Nov 03 20:34:40.071208 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:40.070740 4918 scope.go:122] "RemoveContainer" containerID="1aba7bd51ef0926c55f9b694d24bb37be6e60f8f1a23ee86900ff18f22fe2369" Nov 03 20:34:40.095643 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:40.095614 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz"] Nov 03 20:34:40.100492 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:40.100466 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-drchz"] Nov 03 20:34:40.823212 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:40.820634 4918 patch_prober.go:29] interesting pod/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 20:34:40.823212 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:40.820703 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" podUID="942fc70c-abe0-4988-b6f9-c9babff54817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 20:34:41.216731 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:41.216644 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5120ad5e-81f7-43c1-9e1d-f0407339bb0c" path="/var/lib/kubelet/pods/5120ad5e-81f7-43c1-9e1d-f0407339bb0c/volumes" Nov 03 20:34:41.460340 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:41.460304 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj"] Nov 03 20:34:41.823249 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:41.823217 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:41.823737 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:41.823718 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:41.824561 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:41.824527 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:34:42.727686 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.727656 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:34:42.731351 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.731326 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-token\" (UniqueName: \"kubernetes.io/projected/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-token\") pod \"3e465e5f-cb99-475a-848a-a182ceccc56e\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " Nov 03 20:34:42.731469 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.731380 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e465e5f-cb99-475a-848a-a182ceccc56e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e465e5f-cb99-475a-848a-a182ceccc56e-istiod-ca-cert\") pod \"3e465e5f-cb99-475a-848a-a182ceccc56e\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " Nov 03 20:34:42.731469 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.731431 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-data\") pod \"3e465e5f-cb99-475a-848a-a182ceccc56e\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " Nov 03 20:34:42.731469 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.731459 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-envoy\") pod \"3e465e5f-cb99-475a-848a-a182ceccc56e\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " Nov 03 20:34:42.731657 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.731490 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-workload-certs\") pod \"3e465e5f-cb99-475a-848a-a182ceccc56e\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " Nov 03 20:34:42.731657 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.731550 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-podinfo\") pod \"3e465e5f-cb99-475a-848a-a182ceccc56e\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " Nov 03 20:34:42.731657 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.731626 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-credential-socket\") pod \"3e465e5f-cb99-475a-848a-a182ceccc56e\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " Nov 03 20:34:42.731824 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.731655 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e465e5f-cb99-475a-848a-a182ceccc56e-kube-api-access-wwxgg\" (UniqueName: \"kubernetes.io/projected/3e465e5f-cb99-475a-848a-a182ceccc56e-kube-api-access-wwxgg\") pod \"3e465e5f-cb99-475a-848a-a182ceccc56e\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " Nov 03 20:34:42.731824 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.731780 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-data" pod "3e465e5f-cb99-475a-848a-a182ceccc56e" (UID: "3e465e5f-cb99-475a-848a-a182ceccc56e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:42.731925 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.731879 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e465e5f-cb99-475a-848a-a182ceccc56e-istiod-ca-cert" pod "3e465e5f-cb99-475a-848a-a182ceccc56e" (UID: "3e465e5f-cb99-475a-848a-a182ceccc56e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:42.732137 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.732113 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-credential-socket" pod "3e465e5f-cb99-475a-848a-a182ceccc56e" (UID: "3e465e5f-cb99-475a-848a-a182ceccc56e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:42.732227 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.732154 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-workload-certs" pod "3e465e5f-cb99-475a-848a-a182ceccc56e" (UID: "3e465e5f-cb99-475a-848a-a182ceccc56e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:42.732392 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.732365 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:42.732513 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.732394 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e465e5f-cb99-475a-848a-a182ceccc56e-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:42.732513 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.732408 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:42.732513 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.732417 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:42.733586 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.733561 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-token" pod "3e465e5f-cb99-475a-848a-a182ceccc56e" (UID: "3e465e5f-cb99-475a-848a-a182ceccc56e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:42.733823 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.733806 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e465e5f-cb99-475a-848a-a182ceccc56e-kube-api-access-wwxgg" pod "3e465e5f-cb99-475a-848a-a182ceccc56e" (UID: "3e465e5f-cb99-475a-848a-a182ceccc56e"). InnerVolumeSpecName "kube-api-access-wwxgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:42.734108 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.734084 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-envoy" pod "3e465e5f-cb99-475a-848a-a182ceccc56e" (UID: "3e465e5f-cb99-475a-848a-a182ceccc56e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:42.734201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.734094 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-podinfo" pod "3e465e5f-cb99-475a-848a-a182ceccc56e" (UID: "3e465e5f-cb99-475a-848a-a182ceccc56e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:42.833400 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.833365 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-workload-socket\") pod \"3e465e5f-cb99-475a-848a-a182ceccc56e\" (UID: \"3e465e5f-cb99-475a-848a-a182ceccc56e\") " Nov 03 20:34:42.833632 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.833609 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-workload-socket" pod "3e465e5f-cb99-475a-848a-a182ceccc56e" (UID: "3e465e5f-cb99-475a-848a-a182ceccc56e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:42.833759 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.833744 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:42.833797 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.833762 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwxgg\" (UniqueName: \"kubernetes.io/projected/3e465e5f-cb99-475a-848a-a182ceccc56e-kube-api-access-wwxgg\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:42.833797 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.833772 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:42.833797 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.833781 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:42.833797 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:42.833790 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e465e5f-cb99-475a-848a-a182ceccc56e-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:43.096361 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:43.096323 4918 generic.go:356] "Generic (PLEG): container finished" podID="3e465e5f-cb99-475a-848a-a182ceccc56e" containerID="2dad895aa9ccdc55828c60f4e9f663fe0e96535c05dfd56982ea4b27fd257859" exitCode=0 Nov 03 20:34:43.096557 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:43.096385 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" Nov 03 20:34:43.096557 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:43.096410 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" event={"ID":"3e465e5f-cb99-475a-848a-a182ceccc56e","Type":"ContainerDied","Data":"2dad895aa9ccdc55828c60f4e9f663fe0e96535c05dfd56982ea4b27fd257859"} Nov 03 20:34:43.096557 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:43.096514 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc" event={"ID":"3e465e5f-cb99-475a-848a-a182ceccc56e","Type":"ContainerDied","Data":"16c1b6ff78457e80a84adb84bdb717265573f71d8c1b8d6fc206acf4ce17c663"} Nov 03 20:34:43.096690 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:43.096577 4918 scope.go:122] "RemoveContainer" containerID="2dad895aa9ccdc55828c60f4e9f663fe0e96535c05dfd56982ea4b27fd257859" Nov 03 20:34:43.108574 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:43.108554 4918 scope.go:122] "RemoveContainer" containerID="b86ad3a64e81e11414b069a145f97d32fe73d2ef95172bb327f773f85d26732c" Nov 03 20:34:43.119118 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:43.119097 4918 scope.go:122] "RemoveContainer" containerID="2dad895aa9ccdc55828c60f4e9f663fe0e96535c05dfd56982ea4b27fd257859" Nov 03 20:34:43.119775 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:34:43.119738 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dad895aa9ccdc55828c60f4e9f663fe0e96535c05dfd56982ea4b27fd257859\": container with ID starting with 2dad895aa9ccdc55828c60f4e9f663fe0e96535c05dfd56982ea4b27fd257859 not found: ID does not exist" containerID="2dad895aa9ccdc55828c60f4e9f663fe0e96535c05dfd56982ea4b27fd257859" Nov 03 20:34:43.119943 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:43.119788 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dad895aa9ccdc55828c60f4e9f663fe0e96535c05dfd56982ea4b27fd257859"} err="failed to get container status \"2dad895aa9ccdc55828c60f4e9f663fe0e96535c05dfd56982ea4b27fd257859\": rpc error: code = NotFound desc = could not find container \"2dad895aa9ccdc55828c60f4e9f663fe0e96535c05dfd56982ea4b27fd257859\": container with ID starting with 2dad895aa9ccdc55828c60f4e9f663fe0e96535c05dfd56982ea4b27fd257859 not found: ID does not exist" Nov 03 20:34:43.119943 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:43.119815 4918 scope.go:122] "RemoveContainer" containerID="b86ad3a64e81e11414b069a145f97d32fe73d2ef95172bb327f773f85d26732c" Nov 03 20:34:43.120211 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:34:43.120158 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b86ad3a64e81e11414b069a145f97d32fe73d2ef95172bb327f773f85d26732c\": container with ID starting with b86ad3a64e81e11414b069a145f97d32fe73d2ef95172bb327f773f85d26732c not found: ID does not exist" containerID="b86ad3a64e81e11414b069a145f97d32fe73d2ef95172bb327f773f85d26732c" Nov 03 20:34:43.120288 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:43.120223 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b86ad3a64e81e11414b069a145f97d32fe73d2ef95172bb327f773f85d26732c"} err="failed to get container status \"b86ad3a64e81e11414b069a145f97d32fe73d2ef95172bb327f773f85d26732c\": rpc error: code = NotFound desc = could not find container \"b86ad3a64e81e11414b069a145f97d32fe73d2ef95172bb327f773f85d26732c\": container with ID starting with b86ad3a64e81e11414b069a145f97d32fe73d2ef95172bb327f773f85d26732c not found: ID does not exist" Nov 03 20:34:43.122706 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:43.122685 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc"] Nov 03 20:34:43.128541 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:43.128519 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-kzdmc"] Nov 03 20:34:43.214966 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:43.214932 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e465e5f-cb99-475a-848a-a182ceccc56e" path="/var/lib/kubelet/pods/3e465e5f-cb99-475a-848a-a182ceccc56e/volumes" Nov 03 20:34:46.461367 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:46.461311 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj" podUID="0d724b00-97ad-4d6f-85a6-3a273b5042f1" containerName="payload-pre-processing" containerID="cri-o://fb1e582595e217fa0f29700910b824944d09af60ee8ae9d2e711762348d20d6c" gracePeriod=25 Nov 03 20:34:46.736301 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:46.736265 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj" Nov 03 20:34:46.784591 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:46.784556 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d724b00-97ad-4d6f-85a6-3a273b5042f1-kube-api-access-lzzwl\" (UniqueName: \"kubernetes.io/projected/0d724b00-97ad-4d6f-85a6-3a273b5042f1-kube-api-access-lzzwl\") pod \"0d724b00-97ad-4d6f-85a6-3a273b5042f1\" (UID: \"0d724b00-97ad-4d6f-85a6-3a273b5042f1\") " Nov 03 20:34:46.784755 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:46.784611 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d724b00-97ad-4d6f-85a6-3a273b5042f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d724b00-97ad-4d6f-85a6-3a273b5042f1-plugins-config-volume\") pod \"0d724b00-97ad-4d6f-85a6-3a273b5042f1\" (UID: \"0d724b00-97ad-4d6f-85a6-3a273b5042f1\") " Nov 03 20:34:46.785092 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:46.785063 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d724b00-97ad-4d6f-85a6-3a273b5042f1-plugins-config-volume" pod "0d724b00-97ad-4d6f-85a6-3a273b5042f1" (UID: "0d724b00-97ad-4d6f-85a6-3a273b5042f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:46.785400 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:46.785376 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d724b00-97ad-4d6f-85a6-3a273b5042f1-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:46.786946 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:46.786914 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d724b00-97ad-4d6f-85a6-3a273b5042f1-kube-api-access-lzzwl" pod "0d724b00-97ad-4d6f-85a6-3a273b5042f1" (UID: "0d724b00-97ad-4d6f-85a6-3a273b5042f1"). InnerVolumeSpecName "kube-api-access-lzzwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:46.886580 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:46.886539 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzzwl\" (UniqueName: \"kubernetes.io/projected/0d724b00-97ad-4d6f-85a6-3a273b5042f1-kube-api-access-lzzwl\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:47.129405 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:47.129372 4918 generic.go:356] "Generic (PLEG): container finished" podID="0d724b00-97ad-4d6f-85a6-3a273b5042f1" containerID="fb1e582595e217fa0f29700910b824944d09af60ee8ae9d2e711762348d20d6c" exitCode=0 Nov 03 20:34:47.129564 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:47.129419 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj" Nov 03 20:34:47.129564 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:47.129450 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj" event={"ID":"0d724b00-97ad-4d6f-85a6-3a273b5042f1","Type":"ContainerDied","Data":"fb1e582595e217fa0f29700910b824944d09af60ee8ae9d2e711762348d20d6c"} Nov 03 20:34:47.129564 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:47.129505 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj" event={"ID":"0d724b00-97ad-4d6f-85a6-3a273b5042f1","Type":"ContainerDied","Data":"58eacd342baf31ec6fc441bffaf63efd6a58afd5ecf10c9b39f97f1bcd56fc65"} Nov 03 20:34:47.129564 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:47.129527 4918 scope.go:122] "RemoveContainer" containerID="fb1e582595e217fa0f29700910b824944d09af60ee8ae9d2e711762348d20d6c" Nov 03 20:34:47.140084 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:47.140069 4918 scope.go:122] "RemoveContainer" containerID="fb1e582595e217fa0f29700910b824944d09af60ee8ae9d2e711762348d20d6c" Nov 03 20:34:47.140389 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:34:47.140362 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1e582595e217fa0f29700910b824944d09af60ee8ae9d2e711762348d20d6c\": container with ID starting with fb1e582595e217fa0f29700910b824944d09af60ee8ae9d2e711762348d20d6c not found: ID does not exist" containerID="fb1e582595e217fa0f29700910b824944d09af60ee8ae9d2e711762348d20d6c" Nov 03 20:34:47.140456 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:47.140402 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1e582595e217fa0f29700910b824944d09af60ee8ae9d2e711762348d20d6c"} err="failed to get container status \"fb1e582595e217fa0f29700910b824944d09af60ee8ae9d2e711762348d20d6c\": rpc error: code = NotFound desc = could not find container \"fb1e582595e217fa0f29700910b824944d09af60ee8ae9d2e711762348d20d6c\": container with ID starting with fb1e582595e217fa0f29700910b824944d09af60ee8ae9d2e711762348d20d6c not found: ID does not exist" Nov 03 20:34:47.156509 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:47.156482 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj"] Nov 03 20:34:47.160639 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:47.160617 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-1c6a98-757b96bd67-7rqvj"] Nov 03 20:34:47.214230 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:47.214202 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d724b00-97ad-4d6f-85a6-3a273b5042f1" path="/var/lib/kubelet/pods/0d724b00-97ad-4d6f-85a6-3a273b5042f1/volumes" Nov 03 20:34:48.090081 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:48.090046 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f"] Nov 03 20:34:51.950073 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:51.950033 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" containerID="cri-o://92b7691d4dadc42955585f101711b585ff3ac08b5dab1aa884aad18af3a79b7c" gracePeriod=45 Nov 03 20:34:52.171019 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.170980 4918 generic.go:356] "Generic (PLEG): container finished" podID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerID="92b7691d4dadc42955585f101711b585ff3ac08b5dab1aa884aad18af3a79b7c" exitCode=0 Nov 03 20:34:52.171244 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.171047 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" event={"ID":"bad512a6-de2f-428d-a37a-e56eb56e15f6","Type":"ContainerDied","Data":"92b7691d4dadc42955585f101711b585ff3ac08b5dab1aa884aad18af3a79b7c"} Nov 03 20:34:52.252930 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.252906 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:34:52.363647 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.363582 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-tmp-dir\") pod \"bad512a6-de2f-428d-a37a-e56eb56e15f6\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " Nov 03 20:34:52.364849 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.363674 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-home\" (UniqueName: \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-home\") pod \"bad512a6-de2f-428d-a37a-e56eb56e15f6\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " Nov 03 20:34:52.364849 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.363718 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-dshm\") pod \"bad512a6-de2f-428d-a37a-e56eb56e15f6\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " Nov 03 20:34:52.364849 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.363766 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bad512a6-de2f-428d-a37a-e56eb56e15f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/bad512a6-de2f-428d-a37a-e56eb56e15f6-tls-certs\") pod \"bad512a6-de2f-428d-a37a-e56eb56e15f6\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " Nov 03 20:34:52.364849 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.363806 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad512a6-de2f-428d-a37a-e56eb56e15f6-kube-api-access-t64sp\" (UniqueName: \"kubernetes.io/projected/bad512a6-de2f-428d-a37a-e56eb56e15f6-kube-api-access-t64sp\") pod \"bad512a6-de2f-428d-a37a-e56eb56e15f6\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " Nov 03 20:34:52.364849 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.363980 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-model-cache\") pod \"bad512a6-de2f-428d-a37a-e56eb56e15f6\" (UID: \"bad512a6-de2f-428d-a37a-e56eb56e15f6\") " Nov 03 20:34:52.364849 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.364820 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-model-cache" pod "bad512a6-de2f-428d-a37a-e56eb56e15f6" (UID: "bad512a6-de2f-428d-a37a-e56eb56e15f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:52.365222 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.364977 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-home" pod "bad512a6-de2f-428d-a37a-e56eb56e15f6" (UID: "bad512a6-de2f-428d-a37a-e56eb56e15f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:52.365222 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.365039 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-tmp-dir" pod "bad512a6-de2f-428d-a37a-e56eb56e15f6" (UID: "bad512a6-de2f-428d-a37a-e56eb56e15f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:52.375261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.375226 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-dshm" pod "bad512a6-de2f-428d-a37a-e56eb56e15f6" (UID: "bad512a6-de2f-428d-a37a-e56eb56e15f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:52.375413 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.375384 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bad512a6-de2f-428d-a37a-e56eb56e15f6-tls-certs" pod "bad512a6-de2f-428d-a37a-e56eb56e15f6" (UID: "bad512a6-de2f-428d-a37a-e56eb56e15f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:52.375551 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.375531 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad512a6-de2f-428d-a37a-e56eb56e15f6-kube-api-access-t64sp" pod "bad512a6-de2f-428d-a37a-e56eb56e15f6" (UID: "bad512a6-de2f-428d-a37a-e56eb56e15f6"). InnerVolumeSpecName "kube-api-access-t64sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:52.465645 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.465609 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-model-cache\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.465645 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.465645 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-tmp-dir\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.465842 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.465655 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-home\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.465842 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.465663 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bad512a6-de2f-428d-a37a-e56eb56e15f6-dshm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.465842 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.465672 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bad512a6-de2f-428d-a37a-e56eb56e15f6-tls-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.465842 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:52.465682 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t64sp\" (UniqueName: \"kubernetes.io/projected/bad512a6-de2f-428d-a37a-e56eb56e15f6-kube-api-access-t64sp\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:53.090688 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:53.090640 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f" podUID="38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c" containerName="payload-processing" containerID="cri-o://46a44c88a6578ce74cb13d0a6bde3e69ac8244fc33008061067dc551423581e5" gracePeriod=25 Nov 03 20:34:53.179708 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:53.179678 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" Nov 03 20:34:53.179896 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:53.179710 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66" event={"ID":"bad512a6-de2f-428d-a37a-e56eb56e15f6","Type":"ContainerDied","Data":"b44bda2fcc318d8691786342e7b633e5279159bea2dfcafe06306373ade94837"} Nov 03 20:34:53.179896 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:53.179772 4918 scope.go:122] "RemoveContainer" containerID="92b7691d4dadc42955585f101711b585ff3ac08b5dab1aa884aad18af3a79b7c" Nov 03 20:34:53.204687 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:53.204660 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66"] Nov 03 20:34:53.215899 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:53.215874 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/premium-w6-d412cf-kserve-567868994b-hrm66"] Nov 03 20:34:54.277923 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:54.277877 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f" podUID="38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:34:54.587230 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:54.587194 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk"] Nov 03 20:34:54.587667 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:54.587637 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" podUID="935c467f-7d4e-4ea6-9403-08b0079f774c" containerName="istio-proxy" containerID="cri-o://474619fc9ed4e6b491a6dcbca7b333cdf2a3a4ac18fee2b774e197be58f0b777" gracePeriod=30 Nov 03 20:34:55.232700 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:55.232660 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" path="/var/lib/kubelet/pods/bad512a6-de2f-428d-a37a-e56eb56e15f6/volumes" Nov 03 20:34:57.531332 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:57.531300 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv"] Nov 03 20:34:58.208109 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:34:58.208069 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:34:58.658474 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:58.658441 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg"] Nov 03 20:34:58.675924 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:58.675894 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf"] Nov 03 20:34:59.854311 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:59.854235 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:34:59.980284 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:59.980242 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-data\") pod \"935c467f-7d4e-4ea6-9403-08b0079f774c\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " Nov 03 20:34:59.980468 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:59.980297 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-envoy\") pod \"935c467f-7d4e-4ea6-9403-08b0079f774c\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " Nov 03 20:34:59.980468 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:59.980329 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-token\" (UniqueName: \"kubernetes.io/projected/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-token\") pod \"935c467f-7d4e-4ea6-9403-08b0079f774c\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " Nov 03 20:34:59.980604 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:59.980545 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-data" pod "935c467f-7d4e-4ea6-9403-08b0079f774c" (UID: "935c467f-7d4e-4ea6-9403-08b0079f774c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:59.980841 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:59.980811 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935c467f-7d4e-4ea6-9403-08b0079f774c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/935c467f-7d4e-4ea6-9403-08b0079f774c-istiod-ca-cert\") pod \"935c467f-7d4e-4ea6-9403-08b0079f774c\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " Nov 03 20:34:59.980941 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:59.980888 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-credential-socket\") pod \"935c467f-7d4e-4ea6-9403-08b0079f774c\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " Nov 03 20:34:59.981005 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:59.980947 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-podinfo\") pod \"935c467f-7d4e-4ea6-9403-08b0079f774c\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " Nov 03 20:34:59.981005 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:59.980980 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-workload-certs\") pod \"935c467f-7d4e-4ea6-9403-08b0079f774c\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " Nov 03 20:34:59.981115 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:59.981011 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/935c467f-7d4e-4ea6-9403-08b0079f774c-kube-api-access-95h7x\" (UniqueName: \"kubernetes.io/projected/935c467f-7d4e-4ea6-9403-08b0079f774c-kube-api-access-95h7x\") pod \"935c467f-7d4e-4ea6-9403-08b0079f774c\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " Nov 03 20:34:59.981115 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:59.981106 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-workload-socket\") pod \"935c467f-7d4e-4ea6-9403-08b0079f774c\" (UID: \"935c467f-7d4e-4ea6-9403-08b0079f774c\") " Nov 03 20:34:59.981633 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:59.981612 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:59.981911 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:59.981888 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-workload-socket" pod "935c467f-7d4e-4ea6-9403-08b0079f774c" (UID: "935c467f-7d4e-4ea6-9403-08b0079f774c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:59.981911 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:59.981887 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935c467f-7d4e-4ea6-9403-08b0079f774c-istiod-ca-cert" pod "935c467f-7d4e-4ea6-9403-08b0079f774c" (UID: "935c467f-7d4e-4ea6-9403-08b0079f774c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:59.982061 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:59.981892 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-credential-socket" pod "935c467f-7d4e-4ea6-9403-08b0079f774c" (UID: "935c467f-7d4e-4ea6-9403-08b0079f774c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:59.982192 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:59.982106 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-workload-certs" pod "935c467f-7d4e-4ea6-9403-08b0079f774c" (UID: "935c467f-7d4e-4ea6-9403-08b0079f774c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:59.984167 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:59.984140 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-envoy" pod "935c467f-7d4e-4ea6-9403-08b0079f774c" (UID: "935c467f-7d4e-4ea6-9403-08b0079f774c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:59.985552 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:59.985529 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-token" pod "935c467f-7d4e-4ea6-9403-08b0079f774c" (UID: "935c467f-7d4e-4ea6-9403-08b0079f774c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:59.990638 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:59.990615 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-podinfo" pod "935c467f-7d4e-4ea6-9403-08b0079f774c" (UID: "935c467f-7d4e-4ea6-9403-08b0079f774c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:59.992344 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:34:59.992317 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935c467f-7d4e-4ea6-9403-08b0079f774c-kube-api-access-95h7x" pod "935c467f-7d4e-4ea6-9403-08b0079f774c" (UID: "935c467f-7d4e-4ea6-9403-08b0079f774c"). InnerVolumeSpecName "kube-api-access-95h7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:00.082407 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:00.082372 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/935c467f-7d4e-4ea6-9403-08b0079f774c-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.082407 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:00.082405 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.082627 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:00.082415 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.082627 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:00.082424 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.082627 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:00.082433 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95h7x\" (UniqueName: \"kubernetes.io/projected/935c467f-7d4e-4ea6-9403-08b0079f774c-kube-api-access-95h7x\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.082627 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:00.082443 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.082627 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:00.082452 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.082627 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:00.082460 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/935c467f-7d4e-4ea6-9403-08b0079f774c-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.229042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:00.228946 4918 generic.go:356] "Generic (PLEG): container finished" podID="935c467f-7d4e-4ea6-9403-08b0079f774c" containerID="474619fc9ed4e6b491a6dcbca7b333cdf2a3a4ac18fee2b774e197be58f0b777" exitCode=0 Nov 03 20:35:00.229042 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:00.229014 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" Nov 03 20:35:00.229309 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:00.229038 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" event={"ID":"935c467f-7d4e-4ea6-9403-08b0079f774c","Type":"ContainerDied","Data":"474619fc9ed4e6b491a6dcbca7b333cdf2a3a4ac18fee2b774e197be58f0b777"} Nov 03 20:35:00.229309 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:00.229096 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk" event={"ID":"935c467f-7d4e-4ea6-9403-08b0079f774c","Type":"ContainerDied","Data":"ada934db5a44a8dbb45dc88b4818e2d8d7ecc466d707f32b944197b3ad8aa529"} Nov 03 20:35:00.229309 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:00.229132 4918 scope.go:122] "RemoveContainer" containerID="474619fc9ed4e6b491a6dcbca7b333cdf2a3a4ac18fee2b774e197be58f0b777" Nov 03 20:35:00.256138 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:00.256110 4918 scope.go:122] "RemoveContainer" containerID="474619fc9ed4e6b491a6dcbca7b333cdf2a3a4ac18fee2b774e197be58f0b777" Nov 03 20:35:00.256610 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:35:00.256589 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"474619fc9ed4e6b491a6dcbca7b333cdf2a3a4ac18fee2b774e197be58f0b777\": container with ID starting with 474619fc9ed4e6b491a6dcbca7b333cdf2a3a4ac18fee2b774e197be58f0b777 not found: ID does not exist" containerID="474619fc9ed4e6b491a6dcbca7b333cdf2a3a4ac18fee2b774e197be58f0b777" Nov 03 20:35:00.256678 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:00.256621 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474619fc9ed4e6b491a6dcbca7b333cdf2a3a4ac18fee2b774e197be58f0b777"} err="failed to get container status \"474619fc9ed4e6b491a6dcbca7b333cdf2a3a4ac18fee2b774e197be58f0b777\": rpc error: code = NotFound desc = could not find container \"474619fc9ed4e6b491a6dcbca7b333cdf2a3a4ac18fee2b774e197be58f0b777\": container with ID starting with 474619fc9ed4e6b491a6dcbca7b333cdf2a3a4ac18fee2b774e197be58f0b777 not found: ID does not exist" Nov 03 20:35:00.268153 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:00.268126 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk"] Nov 03 20:35:00.274443 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:00.274421 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-fdrfk"] Nov 03 20:35:01.218361 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:01.218321 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="935c467f-7d4e-4ea6-9403-08b0079f774c" path="/var/lib/kubelet/pods/935c467f-7d4e-4ea6-9403-08b0079f774c/volumes" Nov 03 20:35:04.282691 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:04.282644 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f" podUID="38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:35:05.455695 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:05.455653 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b"] Nov 03 20:35:05.456623 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:05.456564 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" podUID="a1c85e9e-0e22-4860-8859-b36713dcbcfc" containerName="maas-api" containerID="cri-o://daa8d8ad303224286c6790ba7b42461e7d32118ccd6c0846a53127f3cbbf5882" gracePeriod=30 Nov 03 20:35:05.815279 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:05.815243 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" Nov 03 20:35:05.829003 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:05.828968 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1c85e9e-0e22-4860-8859-b36713dcbcfc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1c85e9e-0e22-4860-8859-b36713dcbcfc-maas-api-tls\") pod \"a1c85e9e-0e22-4860-8859-b36713dcbcfc\" (UID: \"a1c85e9e-0e22-4860-8859-b36713dcbcfc\") " Nov 03 20:35:05.829192 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:05.829131 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1c85e9e-0e22-4860-8859-b36713dcbcfc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1c85e9e-0e22-4860-8859-b36713dcbcfc-metrics-certs\") pod \"a1c85e9e-0e22-4860-8859-b36713dcbcfc\" (UID: \"a1c85e9e-0e22-4860-8859-b36713dcbcfc\") " Nov 03 20:35:05.829192 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:05.829159 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c85e9e-0e22-4860-8859-b36713dcbcfc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c85e9e-0e22-4860-8859-b36713dcbcfc-openshift-service-ca\") pod \"a1c85e9e-0e22-4860-8859-b36713dcbcfc\" (UID: \"a1c85e9e-0e22-4860-8859-b36713dcbcfc\") " Nov 03 20:35:05.829325 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:05.829223 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c85e9e-0e22-4860-8859-b36713dcbcfc-kube-api-access-bxftx\" (UniqueName: \"kubernetes.io/projected/a1c85e9e-0e22-4860-8859-b36713dcbcfc-kube-api-access-bxftx\") pod \"a1c85e9e-0e22-4860-8859-b36713dcbcfc\" (UID: \"a1c85e9e-0e22-4860-8859-b36713dcbcfc\") " Nov 03 20:35:05.829584 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:05.829550 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c85e9e-0e22-4860-8859-b36713dcbcfc-openshift-service-ca" pod "a1c85e9e-0e22-4860-8859-b36713dcbcfc" (UID: "a1c85e9e-0e22-4860-8859-b36713dcbcfc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:05.829764 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:05.829740 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c85e9e-0e22-4860-8859-b36713dcbcfc-openshift-service-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.831544 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:05.831512 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c85e9e-0e22-4860-8859-b36713dcbcfc-kube-api-access-bxftx" pod "a1c85e9e-0e22-4860-8859-b36713dcbcfc" (UID: "a1c85e9e-0e22-4860-8859-b36713dcbcfc"). InnerVolumeSpecName "kube-api-access-bxftx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:05.831804 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:05.831774 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1c85e9e-0e22-4860-8859-b36713dcbcfc-metrics-certs" pod "a1c85e9e-0e22-4860-8859-b36713dcbcfc" (UID: "a1c85e9e-0e22-4860-8859-b36713dcbcfc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:05.831804 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:05.831786 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1c85e9e-0e22-4860-8859-b36713dcbcfc-maas-api-tls" pod "a1c85e9e-0e22-4860-8859-b36713dcbcfc" (UID: "a1c85e9e-0e22-4860-8859-b36713dcbcfc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:05.931069 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:05.931022 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1c85e9e-0e22-4860-8859-b36713dcbcfc-metrics-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.931069 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:05.931066 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxftx\" (UniqueName: \"kubernetes.io/projected/a1c85e9e-0e22-4860-8859-b36713dcbcfc-kube-api-access-bxftx\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.931356 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:05.931085 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1c85e9e-0e22-4860-8859-b36713dcbcfc-maas-api-tls\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.989512 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:05.989475 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw"] Nov 03 20:35:06.274037 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:06.273999 4918 generic.go:356] "Generic (PLEG): container finished" podID="a1c85e9e-0e22-4860-8859-b36713dcbcfc" containerID="daa8d8ad303224286c6790ba7b42461e7d32118ccd6c0846a53127f3cbbf5882" exitCode=0 Nov 03 20:35:06.274227 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:06.274041 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" Nov 03 20:35:06.274227 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:06.274078 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" event={"ID":"a1c85e9e-0e22-4860-8859-b36713dcbcfc","Type":"ContainerDied","Data":"daa8d8ad303224286c6790ba7b42461e7d32118ccd6c0846a53127f3cbbf5882"} Nov 03 20:35:06.274227 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:06.274133 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b" event={"ID":"a1c85e9e-0e22-4860-8859-b36713dcbcfc","Type":"ContainerDied","Data":"6e156a9bcb81201ef37915c583ca540a6168a28561a033d350aa01413f0db069"} Nov 03 20:35:06.274227 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:06.274164 4918 scope.go:122] "RemoveContainer" containerID="daa8d8ad303224286c6790ba7b42461e7d32118ccd6c0846a53127f3cbbf5882" Nov 03 20:35:06.285334 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:06.285308 4918 scope.go:122] "RemoveContainer" containerID="daa8d8ad303224286c6790ba7b42461e7d32118ccd6c0846a53127f3cbbf5882" Nov 03 20:35:06.285646 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:35:06.285625 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa8d8ad303224286c6790ba7b42461e7d32118ccd6c0846a53127f3cbbf5882\": container with ID starting with daa8d8ad303224286c6790ba7b42461e7d32118ccd6c0846a53127f3cbbf5882 not found: ID does not exist" containerID="daa8d8ad303224286c6790ba7b42461e7d32118ccd6c0846a53127f3cbbf5882" Nov 03 20:35:06.285734 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:06.285656 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa8d8ad303224286c6790ba7b42461e7d32118ccd6c0846a53127f3cbbf5882"} err="failed to get container status \"daa8d8ad303224286c6790ba7b42461e7d32118ccd6c0846a53127f3cbbf5882\": rpc error: code = NotFound desc = could not find container \"daa8d8ad303224286c6790ba7b42461e7d32118ccd6c0846a53127f3cbbf5882\": container with ID starting with daa8d8ad303224286c6790ba7b42461e7d32118ccd6c0846a53127f3cbbf5882 not found: ID does not exist" Nov 03 20:35:06.299200 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:06.299160 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b"] Nov 03 20:35:06.305038 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:06.305014 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-d412cf-5f58f958fc-ntw7b"] Nov 03 20:35:07.215890 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:07.215857 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c85e9e-0e22-4860-8859-b36713dcbcfc" path="/var/lib/kubelet/pods/a1c85e9e-0e22-4860-8859-b36713dcbcfc/volumes" Nov 03 20:35:11.340015 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:11.339920 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp"] Nov 03 20:35:12.552353 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:12.552290 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" podUID="54a415e5-f853-4579-9f3f-0c3c7a9216e5" containerName="main" containerID="cri-o://35e5a6417f38e1beeb118049bb5e6fff6b066320581db3af8688bef398ba2e62" gracePeriod=45 Nov 03 20:35:12.827797 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:12.827765 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:35:12.918164 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:12.918134 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-model-cache\") pod \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " Nov 03 20:35:12.918356 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:12.918200 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-home\" (UniqueName: \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-home\") pod \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " Nov 03 20:35:12.918356 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:12.918264 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-tmp-dir\") pod \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " Nov 03 20:35:12.918356 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:12.918332 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-dshm\") pod \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " Nov 03 20:35:12.918527 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:12.918373 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54a415e5-f853-4579-9f3f-0c3c7a9216e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/54a415e5-f853-4579-9f3f-0c3c7a9216e5-tls-certs\") pod \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " Nov 03 20:35:12.918527 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:12.918407 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a415e5-f853-4579-9f3f-0c3c7a9216e5-kube-api-access-6hqlb\" (UniqueName: \"kubernetes.io/projected/54a415e5-f853-4579-9f3f-0c3c7a9216e5-kube-api-access-6hqlb\") pod \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\" (UID: \"54a415e5-f853-4579-9f3f-0c3c7a9216e5\") " Nov 03 20:35:12.918527 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:12.918397 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-model-cache" pod "54a415e5-f853-4579-9f3f-0c3c7a9216e5" (UID: "54a415e5-f853-4579-9f3f-0c3c7a9216e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:12.918527 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:12.918457 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-home" pod "54a415e5-f853-4579-9f3f-0c3c7a9216e5" (UID: "54a415e5-f853-4579-9f3f-0c3c7a9216e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:12.918698 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:12.918568 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-tmp-dir" pod "54a415e5-f853-4579-9f3f-0c3c7a9216e5" (UID: "54a415e5-f853-4579-9f3f-0c3c7a9216e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:12.918917 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:12.918896 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-model-cache\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:12.918986 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:12.918920 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-home\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:12.918986 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:12.918930 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-tmp-dir\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:12.920725 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:12.920698 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a415e5-f853-4579-9f3f-0c3c7a9216e5-tls-certs" pod "54a415e5-f853-4579-9f3f-0c3c7a9216e5" (UID: "54a415e5-f853-4579-9f3f-0c3c7a9216e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:12.920820 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:12.920698 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-dshm" pod "54a415e5-f853-4579-9f3f-0c3c7a9216e5" (UID: "54a415e5-f853-4579-9f3f-0c3c7a9216e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:12.920820 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:12.920705 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a415e5-f853-4579-9f3f-0c3c7a9216e5-kube-api-access-6hqlb" pod "54a415e5-f853-4579-9f3f-0c3c7a9216e5" (UID: "54a415e5-f853-4579-9f3f-0c3c7a9216e5"). InnerVolumeSpecName "kube-api-access-6hqlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:13.020532 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.020499 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54a415e5-f853-4579-9f3f-0c3c7a9216e5-dshm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:13.020532 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.020530 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54a415e5-f853-4579-9f3f-0c3c7a9216e5-tls-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:13.020738 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.020544 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hqlb\" (UniqueName: \"kubernetes.io/projected/54a415e5-f853-4579-9f3f-0c3c7a9216e5-kube-api-access-6hqlb\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:13.334773 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.334737 4918 generic.go:356] "Generic (PLEG): container finished" podID="38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c" containerID="46a44c88a6578ce74cb13d0a6bde3e69ac8244fc33008061067dc551423581e5" exitCode=0 Nov 03 20:35:13.335118 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.335089 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f" event={"ID":"38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c","Type":"ContainerDied","Data":"46a44c88a6578ce74cb13d0a6bde3e69ac8244fc33008061067dc551423581e5"} Nov 03 20:35:13.337034 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.337010 4918 generic.go:356] "Generic (PLEG): container finished" podID="54a415e5-f853-4579-9f3f-0c3c7a9216e5" containerID="35e5a6417f38e1beeb118049bb5e6fff6b066320581db3af8688bef398ba2e62" exitCode=0 Nov 03 20:35:13.337262 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.337244 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" event={"ID":"54a415e5-f853-4579-9f3f-0c3c7a9216e5","Type":"ContainerDied","Data":"35e5a6417f38e1beeb118049bb5e6fff6b066320581db3af8688bef398ba2e62"} Nov 03 20:35:13.337392 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.337377 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" event={"ID":"54a415e5-f853-4579-9f3f-0c3c7a9216e5","Type":"ContainerDied","Data":"550172d04605202789dc715958a0097ed69ae10d8a27dc04dddf1cfc97e04417"} Nov 03 20:35:13.337493 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.337481 4918 scope.go:122] "RemoveContainer" containerID="35e5a6417f38e1beeb118049bb5e6fff6b066320581db3af8688bef398ba2e62" Nov 03 20:35:13.337715 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.337701 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv" Nov 03 20:35:13.353920 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.353901 4918 scope.go:122] "RemoveContainer" containerID="35e5a6417f38e1beeb118049bb5e6fff6b066320581db3af8688bef398ba2e62" Nov 03 20:35:13.354267 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:35:13.354241 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e5a6417f38e1beeb118049bb5e6fff6b066320581db3af8688bef398ba2e62\": container with ID starting with 35e5a6417f38e1beeb118049bb5e6fff6b066320581db3af8688bef398ba2e62 not found: ID does not exist" containerID="35e5a6417f38e1beeb118049bb5e6fff6b066320581db3af8688bef398ba2e62" Nov 03 20:35:13.354357 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.354277 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e5a6417f38e1beeb118049bb5e6fff6b066320581db3af8688bef398ba2e62"} err="failed to get container status \"35e5a6417f38e1beeb118049bb5e6fff6b066320581db3af8688bef398ba2e62\": rpc error: code = NotFound desc = could not find container \"35e5a6417f38e1beeb118049bb5e6fff6b066320581db3af8688bef398ba2e62\": container with ID starting with 35e5a6417f38e1beeb118049bb5e6fff6b066320581db3af8688bef398ba2e62 not found: ID does not exist" Nov 03 20:35:13.363547 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.363518 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv"] Nov 03 20:35:13.365909 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.365883 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-45e569/test-model-45e569-kserve-849bdb9979-nf7cv"] Nov 03 20:35:13.387387 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.387363 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f" Nov 03 20:35:13.434939 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.434753 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c-plugins-config-volume\") pod \"38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c\" (UID: \"38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c\") " Nov 03 20:35:13.436431 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.436394 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c-kube-api-access-zwfkn\" (UniqueName: \"kubernetes.io/projected/38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c-kube-api-access-zwfkn\") pod \"38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c\" (UID: \"38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c\") " Nov 03 20:35:13.450379 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.437737 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c-plugins-config-volume" pod "38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c" (UID: "38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:13.450379 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.440759 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c-kube-api-access-zwfkn" pod "38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c" (UID: "38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c"). InnerVolumeSpecName "kube-api-access-zwfkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:13.538117 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.538080 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwfkn\" (UniqueName: \"kubernetes.io/projected/38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c-kube-api-access-zwfkn\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:13.538117 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.538112 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:13.710083 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.709815 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" containerID="cri-o://a147f149cc7cc82ba8a418b1de868678afa352b3645d18484e22d155fc7f6e35" gracePeriod=45 Nov 03 20:35:13.710083 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.709889 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" containerID="cri-o://4546ad176dac149c56fc9a63b563e1a918446a8108123a0e6b7723a8be3d4818" gracePeriod=45 Nov 03 20:35:13.745461 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:13.745377 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": dial tcp 10.132.0.56:8000: connect: connection refused" Nov 03 20:35:13.777132 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:35:13.777104 4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod812fc025_3310_47d1_80dd_72b73ef4494e.slice/crio-conmon-a147f149cc7cc82ba8a418b1de868678afa352b3645d18484e22d155fc7f6e35.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:35:14.038272 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.037920 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:35:14.077228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.077165 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:35:14.149068 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.149030 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-home\" (UniqueName: \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-home\") pod \"189c26eb-8160-4721-9775-f27379339f50\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " Nov 03 20:35:14.149263 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.149105 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-home\" (UniqueName: \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-home\") pod \"812fc025-3310-47d1-80dd-72b73ef4494e\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " Nov 03 20:35:14.149263 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.149238 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/812fc025-3310-47d1-80dd-72b73ef4494e-tls-certs\" (UniqueName: \"kubernetes.io/secret/812fc025-3310-47d1-80dd-72b73ef4494e-tls-certs\") pod \"812fc025-3310-47d1-80dd-72b73ef4494e\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " Nov 03 20:35:14.149388 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.149271 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-home" pod "189c26eb-8160-4721-9775-f27379339f50" (UID: "189c26eb-8160-4721-9775-f27379339f50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:14.149388 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.149291 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/189c26eb-8160-4721-9775-f27379339f50-tls-certs\" (UniqueName: \"kubernetes.io/secret/189c26eb-8160-4721-9775-f27379339f50-tls-certs\") pod \"189c26eb-8160-4721-9775-f27379339f50\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " Nov 03 20:35:14.149388 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.149323 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-model-cache\") pod \"812fc025-3310-47d1-80dd-72b73ef4494e\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " Nov 03 20:35:14.149388 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.149364 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/812fc025-3310-47d1-80dd-72b73ef4494e-kube-api-access-w77bd\" (UniqueName: \"kubernetes.io/projected/812fc025-3310-47d1-80dd-72b73ef4494e-kube-api-access-w77bd\") pod \"812fc025-3310-47d1-80dd-72b73ef4494e\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " Nov 03 20:35:14.149597 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.149393 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-home" pod "812fc025-3310-47d1-80dd-72b73ef4494e" (UID: "812fc025-3310-47d1-80dd-72b73ef4494e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:14.149597 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.149454 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-model-cache\") pod \"189c26eb-8160-4721-9775-f27379339f50\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " Nov 03 20:35:14.149597 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.149488 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-dshm\") pod \"812fc025-3310-47d1-80dd-72b73ef4494e\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " Nov 03 20:35:14.149597 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.149529 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-tmp-dir\") pod \"189c26eb-8160-4721-9775-f27379339f50\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " Nov 03 20:35:14.149597 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.149537 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-model-cache" pod "812fc025-3310-47d1-80dd-72b73ef4494e" (UID: "812fc025-3310-47d1-80dd-72b73ef4494e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:14.149872 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.149616 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-tmp-dir\") pod \"812fc025-3310-47d1-80dd-72b73ef4494e\" (UID: \"812fc025-3310-47d1-80dd-72b73ef4494e\") " Nov 03 20:35:14.149872 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.149676 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189c26eb-8160-4721-9775-f27379339f50-kube-api-access-v88vt\" (UniqueName: \"kubernetes.io/projected/189c26eb-8160-4721-9775-f27379339f50-kube-api-access-v88vt\") pod \"189c26eb-8160-4721-9775-f27379339f50\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " Nov 03 20:35:14.149872 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.149719 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-dshm\" (UniqueName: \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-dshm\") pod \"189c26eb-8160-4721-9775-f27379339f50\" (UID: \"189c26eb-8160-4721-9775-f27379339f50\") " Nov 03 20:35:14.150029 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.149893 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-tmp-dir" pod "812fc025-3310-47d1-80dd-72b73ef4494e" (UID: "812fc025-3310-47d1-80dd-72b73ef4494e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:14.150029 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.149911 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-tmp-dir" pod "189c26eb-8160-4721-9775-f27379339f50" (UID: "189c26eb-8160-4721-9775-f27379339f50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:14.150313 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.150261 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-model-cache" pod "189c26eb-8160-4721-9775-f27379339f50" (UID: "189c26eb-8160-4721-9775-f27379339f50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:14.150455 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.150319 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-tmp-dir\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.150455 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.150343 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-tmp-dir\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.150455 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.150358 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-home\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.150455 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.150371 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-home\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.150455 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.150384 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-model-cache\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.152370 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.152340 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/812fc025-3310-47d1-80dd-72b73ef4494e-tls-certs" pod "812fc025-3310-47d1-80dd-72b73ef4494e" (UID: "812fc025-3310-47d1-80dd-72b73ef4494e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:14.152370 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.152355 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-dshm" pod "812fc025-3310-47d1-80dd-72b73ef4494e" (UID: "812fc025-3310-47d1-80dd-72b73ef4494e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:14.152370 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.152363 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189c26eb-8160-4721-9775-f27379339f50-kube-api-access-v88vt" pod "189c26eb-8160-4721-9775-f27379339f50" (UID: "189c26eb-8160-4721-9775-f27379339f50"). InnerVolumeSpecName "kube-api-access-v88vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:14.152542 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.152380 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/189c26eb-8160-4721-9775-f27379339f50-tls-certs" pod "189c26eb-8160-4721-9775-f27379339f50" (UID: "189c26eb-8160-4721-9775-f27379339f50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:14.152593 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.152549 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/812fc025-3310-47d1-80dd-72b73ef4494e-kube-api-access-w77bd" pod "812fc025-3310-47d1-80dd-72b73ef4494e" (UID: "812fc025-3310-47d1-80dd-72b73ef4494e"). InnerVolumeSpecName "kube-api-access-w77bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:14.152631 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.152617 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-dshm" pod "189c26eb-8160-4721-9775-f27379339f50" (UID: "189c26eb-8160-4721-9775-f27379339f50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:14.253270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.252845 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v88vt\" (UniqueName: \"kubernetes.io/projected/189c26eb-8160-4721-9775-f27379339f50-kube-api-access-v88vt\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.253270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.252878 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-dshm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.253270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.252892 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/812fc025-3310-47d1-80dd-72b73ef4494e-tls-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.253270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.252906 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/189c26eb-8160-4721-9775-f27379339f50-tls-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.253270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.252919 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w77bd\" (UniqueName: \"kubernetes.io/projected/812fc025-3310-47d1-80dd-72b73ef4494e-kube-api-access-w77bd\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.253270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.252932 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/189c26eb-8160-4721-9775-f27379339f50-model-cache\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.253270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.252945 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/812fc025-3310-47d1-80dd-72b73ef4494e-dshm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.344879 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.344842 4918 generic.go:356] "Generic (PLEG): container finished" podID="189c26eb-8160-4721-9775-f27379339f50" containerID="4546ad176dac149c56fc9a63b563e1a918446a8108123a0e6b7723a8be3d4818" exitCode=0 Nov 03 20:35:14.345057 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.344898 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" Nov 03 20:35:14.345057 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.344923 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" event={"ID":"189c26eb-8160-4721-9775-f27379339f50","Type":"ContainerDied","Data":"4546ad176dac149c56fc9a63b563e1a918446a8108123a0e6b7723a8be3d4818"} Nov 03 20:35:14.345057 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.344963 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf" event={"ID":"189c26eb-8160-4721-9775-f27379339f50","Type":"ContainerDied","Data":"0b1eab39ee4c220cbf7ec2426d8de09b4760eb0b6a0bf63385037abff597d393"} Nov 03 20:35:14.345057 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.344979 4918 scope.go:122] "RemoveContainer" containerID="4546ad176dac149c56fc9a63b563e1a918446a8108123a0e6b7723a8be3d4818" Nov 03 20:35:14.346350 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.346329 4918 generic.go:356] "Generic (PLEG): container finished" podID="812fc025-3310-47d1-80dd-72b73ef4494e" containerID="a147f149cc7cc82ba8a418b1de868678afa352b3645d18484e22d155fc7f6e35" exitCode=0 Nov 03 20:35:14.346442 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.346363 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" event={"ID":"812fc025-3310-47d1-80dd-72b73ef4494e","Type":"ContainerDied","Data":"a147f149cc7cc82ba8a418b1de868678afa352b3645d18484e22d155fc7f6e35"} Nov 03 20:35:14.346442 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.346386 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" Nov 03 20:35:14.346442 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.346401 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg" event={"ID":"812fc025-3310-47d1-80dd-72b73ef4494e","Type":"ContainerDied","Data":"d2f205963a6c37eb686f7b02efa7118d2afbff09c53f91eb42675a86fe9481d5"} Nov 03 20:35:14.347853 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.347829 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f" event={"ID":"38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c","Type":"ContainerDied","Data":"bda811ec90e31a50acca43dcb844cf5a3238185ad657ec360158964d114627f7"} Nov 03 20:35:14.347988 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.347834 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f" Nov 03 20:35:14.358447 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.358382 4918 scope.go:122] "RemoveContainer" containerID="4546ad176dac149c56fc9a63b563e1a918446a8108123a0e6b7723a8be3d4818" Nov 03 20:35:14.358908 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:35:14.358881 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4546ad176dac149c56fc9a63b563e1a918446a8108123a0e6b7723a8be3d4818\": container with ID starting with 4546ad176dac149c56fc9a63b563e1a918446a8108123a0e6b7723a8be3d4818 not found: ID does not exist" containerID="4546ad176dac149c56fc9a63b563e1a918446a8108123a0e6b7723a8be3d4818" Nov 03 20:35:14.359012 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.358920 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4546ad176dac149c56fc9a63b563e1a918446a8108123a0e6b7723a8be3d4818"} err="failed to get container status \"4546ad176dac149c56fc9a63b563e1a918446a8108123a0e6b7723a8be3d4818\": rpc error: code = NotFound desc = could not find container \"4546ad176dac149c56fc9a63b563e1a918446a8108123a0e6b7723a8be3d4818\": container with ID starting with 4546ad176dac149c56fc9a63b563e1a918446a8108123a0e6b7723a8be3d4818 not found: ID does not exist" Nov 03 20:35:14.359012 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.358944 4918 scope.go:122] "RemoveContainer" containerID="a147f149cc7cc82ba8a418b1de868678afa352b3645d18484e22d155fc7f6e35" Nov 03 20:35:14.369032 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.369012 4918 scope.go:122] "RemoveContainer" containerID="a147f149cc7cc82ba8a418b1de868678afa352b3645d18484e22d155fc7f6e35" Nov 03 20:35:14.369285 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:35:14.369266 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a147f149cc7cc82ba8a418b1de868678afa352b3645d18484e22d155fc7f6e35\": container with ID starting with a147f149cc7cc82ba8a418b1de868678afa352b3645d18484e22d155fc7f6e35 not found: ID does not exist" containerID="a147f149cc7cc82ba8a418b1de868678afa352b3645d18484e22d155fc7f6e35" Nov 03 20:35:14.369350 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.369291 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a147f149cc7cc82ba8a418b1de868678afa352b3645d18484e22d155fc7f6e35"} err="failed to get container status \"a147f149cc7cc82ba8a418b1de868678afa352b3645d18484e22d155fc7f6e35\": rpc error: code = NotFound desc = could not find container \"a147f149cc7cc82ba8a418b1de868678afa352b3645d18484e22d155fc7f6e35\": container with ID starting with a147f149cc7cc82ba8a418b1de868678afa352b3645d18484e22d155fc7f6e35 not found: ID does not exist" Nov 03 20:35:14.369350 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.369307 4918 scope.go:122] "RemoveContainer" containerID="46a44c88a6578ce74cb13d0a6bde3e69ac8244fc33008061067dc551423581e5" Nov 03 20:35:14.376164 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.376142 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf"] Nov 03 20:35:14.380864 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.380845 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/simulator-w4-3aba31-kserve-544c766f44-2lhtf"] Nov 03 20:35:14.392364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.392344 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg"] Nov 03 20:35:14.396363 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.396342 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/distinct-w4-3aba31-kserve-79f5c97c44-hvmqg"] Nov 03 20:35:14.411570 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.411545 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f"] Nov 03 20:35:14.418386 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:14.418366 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a744eb-5d79494c7b-pzs8f"] Nov 03 20:35:15.244209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:15.242248 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189c26eb-8160-4721-9775-f27379339f50" path="/var/lib/kubelet/pods/189c26eb-8160-4721-9775-f27379339f50/volumes" Nov 03 20:35:15.244209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:15.242957 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c" path="/var/lib/kubelet/pods/38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c/volumes" Nov 03 20:35:15.244209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:15.243606 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54a415e5-f853-4579-9f3f-0c3c7a9216e5" path="/var/lib/kubelet/pods/54a415e5-f853-4579-9f3f-0c3c7a9216e5/volumes" Nov 03 20:35:15.244968 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:15.244679 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" path="/var/lib/kubelet/pods/812fc025-3310-47d1-80dd-72b73ef4494e/volumes" Nov 03 20:35:16.341236 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:16.341169 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp" podUID="e2ae2662-2a94-411f-a9e1-a0fb1452f3ae" containerName="payload-pre-processing" containerID="cri-o://4fbab76bf14faf13fdaf355e672491d7013ac602bbeb3c18848fbbc8c21a41fa" gracePeriod=25 Nov 03 20:35:16.609278 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:16.609204 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp" Nov 03 20:35:16.680656 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:16.680624 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2ae2662-2a94-411f-a9e1-a0fb1452f3ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ae2662-2a94-411f-a9e1-a0fb1452f3ae-plugins-config-volume\") pod \"e2ae2662-2a94-411f-a9e1-a0fb1452f3ae\" (UID: \"e2ae2662-2a94-411f-a9e1-a0fb1452f3ae\") " Nov 03 20:35:16.680827 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:16.680687 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2ae2662-2a94-411f-a9e1-a0fb1452f3ae-kube-api-access-z4d92\" (UniqueName: \"kubernetes.io/projected/e2ae2662-2a94-411f-a9e1-a0fb1452f3ae-kube-api-access-z4d92\") pod \"e2ae2662-2a94-411f-a9e1-a0fb1452f3ae\" (UID: \"e2ae2662-2a94-411f-a9e1-a0fb1452f3ae\") " Nov 03 20:35:16.681079 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:16.681052 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ae2662-2a94-411f-a9e1-a0fb1452f3ae-plugins-config-volume" pod "e2ae2662-2a94-411f-a9e1-a0fb1452f3ae" (UID: "e2ae2662-2a94-411f-a9e1-a0fb1452f3ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:16.681218 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:16.681198 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ae2662-2a94-411f-a9e1-a0fb1452f3ae-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:16.683317 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:16.683284 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ae2662-2a94-411f-a9e1-a0fb1452f3ae-kube-api-access-z4d92" pod "e2ae2662-2a94-411f-a9e1-a0fb1452f3ae" (UID: "e2ae2662-2a94-411f-a9e1-a0fb1452f3ae"). InnerVolumeSpecName "kube-api-access-z4d92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:16.782195 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:16.782147 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4d92\" (UniqueName: \"kubernetes.io/projected/e2ae2662-2a94-411f-a9e1-a0fb1452f3ae-kube-api-access-z4d92\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:17.372169 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:17.372135 4918 generic.go:356] "Generic (PLEG): container finished" podID="e2ae2662-2a94-411f-a9e1-a0fb1452f3ae" containerID="4fbab76bf14faf13fdaf355e672491d7013ac602bbeb3c18848fbbc8c21a41fa" exitCode=0 Nov 03 20:35:17.372567 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:17.372205 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp" Nov 03 20:35:17.372567 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:17.372207 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp" event={"ID":"e2ae2662-2a94-411f-a9e1-a0fb1452f3ae","Type":"ContainerDied","Data":"4fbab76bf14faf13fdaf355e672491d7013ac602bbeb3c18848fbbc8c21a41fa"} Nov 03 20:35:17.372567 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:17.372321 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp" event={"ID":"e2ae2662-2a94-411f-a9e1-a0fb1452f3ae","Type":"ContainerDied","Data":"bc40aa39652a4129ec16ffb2bd3b9d5b11489b354b2faa2ad47a1343723b8140"} Nov 03 20:35:17.372567 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:17.372343 4918 scope.go:122] "RemoveContainer" containerID="4fbab76bf14faf13fdaf355e672491d7013ac602bbeb3c18848fbbc8c21a41fa" Nov 03 20:35:17.381862 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:17.381845 4918 scope.go:122] "RemoveContainer" containerID="4fbab76bf14faf13fdaf355e672491d7013ac602bbeb3c18848fbbc8c21a41fa" Nov 03 20:35:17.382110 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:35:17.382090 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fbab76bf14faf13fdaf355e672491d7013ac602bbeb3c18848fbbc8c21a41fa\": container with ID starting with 4fbab76bf14faf13fdaf355e672491d7013ac602bbeb3c18848fbbc8c21a41fa not found: ID does not exist" containerID="4fbab76bf14faf13fdaf355e672491d7013ac602bbeb3c18848fbbc8c21a41fa" Nov 03 20:35:17.382166 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:17.382117 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fbab76bf14faf13fdaf355e672491d7013ac602bbeb3c18848fbbc8c21a41fa"} err="failed to get container status \"4fbab76bf14faf13fdaf355e672491d7013ac602bbeb3c18848fbbc8c21a41fa\": rpc error: code = NotFound desc = could not find container \"4fbab76bf14faf13fdaf355e672491d7013ac602bbeb3c18848fbbc8c21a41fa\": container with ID starting with 4fbab76bf14faf13fdaf355e672491d7013ac602bbeb3c18848fbbc8c21a41fa not found: ID does not exist" Nov 03 20:35:17.391325 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:17.391302 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp"] Nov 03 20:35:17.395245 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:17.395224 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a6f98b-64f49b49f-gk4hp"] Nov 03 20:35:18.639103 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:18.639067 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m"] Nov 03 20:35:19.222607 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:19.222295 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2ae2662-2a94-411f-a9e1-a0fb1452f3ae" path="/var/lib/kubelet/pods/e2ae2662-2a94-411f-a9e1-a0fb1452f3ae/volumes" Nov 03 20:35:19.736346 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:19.736311 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g"] Nov 03 20:35:19.736720 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:19.736494 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" podUID="077732c1-4eda-4ce4-8afe-44e00e632df9" containerName="maas-api" containerID="cri-o://bd1306b94e6ee2c499edebf6519db2670c65e6d43545c8cd9191569bd66f5a84" gracePeriod=30 Nov 03 20:35:19.772542 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:19.772511 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" podUID="077732c1-4eda-4ce4-8afe-44e00e632df9" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.55:8443/health\": dial tcp 10.132.0.55:8443: connect: connection refused" Nov 03 20:35:20.077361 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.077334 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" Nov 03 20:35:20.150646 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.150615 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/077732c1-4eda-4ce4-8afe-44e00e632df9-kube-api-access-bdgz5\" (UniqueName: \"kubernetes.io/projected/077732c1-4eda-4ce4-8afe-44e00e632df9-kube-api-access-bdgz5\") pod \"077732c1-4eda-4ce4-8afe-44e00e632df9\" (UID: \"077732c1-4eda-4ce4-8afe-44e00e632df9\") " Nov 03 20:35:20.150887 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.150689 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/077732c1-4eda-4ce4-8afe-44e00e632df9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/077732c1-4eda-4ce4-8afe-44e00e632df9-metrics-certs\") pod \"077732c1-4eda-4ce4-8afe-44e00e632df9\" (UID: \"077732c1-4eda-4ce4-8afe-44e00e632df9\") " Nov 03 20:35:20.150887 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.150739 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/077732c1-4eda-4ce4-8afe-44e00e632df9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/077732c1-4eda-4ce4-8afe-44e00e632df9-maas-api-tls\") pod \"077732c1-4eda-4ce4-8afe-44e00e632df9\" (UID: \"077732c1-4eda-4ce4-8afe-44e00e632df9\") " Nov 03 20:35:20.150887 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.150865 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/077732c1-4eda-4ce4-8afe-44e00e632df9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/077732c1-4eda-4ce4-8afe-44e00e632df9-openshift-service-ca\") pod \"077732c1-4eda-4ce4-8afe-44e00e632df9\" (UID: \"077732c1-4eda-4ce4-8afe-44e00e632df9\") " Nov 03 20:35:20.151354 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.151299 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/077732c1-4eda-4ce4-8afe-44e00e632df9-openshift-service-ca" pod "077732c1-4eda-4ce4-8afe-44e00e632df9" (UID: "077732c1-4eda-4ce4-8afe-44e00e632df9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:20.153125 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.153102 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/077732c1-4eda-4ce4-8afe-44e00e632df9-maas-api-tls" pod "077732c1-4eda-4ce4-8afe-44e00e632df9" (UID: "077732c1-4eda-4ce4-8afe-44e00e632df9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:20.153306 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.153289 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/077732c1-4eda-4ce4-8afe-44e00e632df9-metrics-certs" pod "077732c1-4eda-4ce4-8afe-44e00e632df9" (UID: "077732c1-4eda-4ce4-8afe-44e00e632df9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:20.153478 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.153461 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/077732c1-4eda-4ce4-8afe-44e00e632df9-kube-api-access-bdgz5" pod "077732c1-4eda-4ce4-8afe-44e00e632df9" (UID: "077732c1-4eda-4ce4-8afe-44e00e632df9"). InnerVolumeSpecName "kube-api-access-bdgz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:20.252713 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.252678 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/077732c1-4eda-4ce4-8afe-44e00e632df9-openshift-service-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:20.252713 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.252707 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdgz5\" (UniqueName: \"kubernetes.io/projected/077732c1-4eda-4ce4-8afe-44e00e632df9-kube-api-access-bdgz5\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:20.252713 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.252719 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/077732c1-4eda-4ce4-8afe-44e00e632df9-metrics-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:20.252963 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.252729 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/077732c1-4eda-4ce4-8afe-44e00e632df9-maas-api-tls\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:20.394035 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.393999 4918 generic.go:356] "Generic (PLEG): container finished" podID="077732c1-4eda-4ce4-8afe-44e00e632df9" containerID="bd1306b94e6ee2c499edebf6519db2670c65e6d43545c8cd9191569bd66f5a84" exitCode=0 Nov 03 20:35:20.394035 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.394037 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" event={"ID":"077732c1-4eda-4ce4-8afe-44e00e632df9","Type":"ContainerDied","Data":"bd1306b94e6ee2c499edebf6519db2670c65e6d43545c8cd9191569bd66f5a84"} Nov 03 20:35:20.394314 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.394049 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" Nov 03 20:35:20.394314 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.394080 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g" event={"ID":"077732c1-4eda-4ce4-8afe-44e00e632df9","Type":"ContainerDied","Data":"276b3e0163b662e01d20c51c2adbfbb518028b395756eccf5961216ce5440e56"} Nov 03 20:35:20.394314 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.394104 4918 scope.go:122] "RemoveContainer" containerID="bd1306b94e6ee2c499edebf6519db2670c65e6d43545c8cd9191569bd66f5a84" Nov 03 20:35:20.404229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.404208 4918 scope.go:122] "RemoveContainer" containerID="bd1306b94e6ee2c499edebf6519db2670c65e6d43545c8cd9191569bd66f5a84" Nov 03 20:35:20.404499 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:35:20.404480 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd1306b94e6ee2c499edebf6519db2670c65e6d43545c8cd9191569bd66f5a84\": container with ID starting with bd1306b94e6ee2c499edebf6519db2670c65e6d43545c8cd9191569bd66f5a84 not found: ID does not exist" containerID="bd1306b94e6ee2c499edebf6519db2670c65e6d43545c8cd9191569bd66f5a84" Nov 03 20:35:20.404576 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.404511 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd1306b94e6ee2c499edebf6519db2670c65e6d43545c8cd9191569bd66f5a84"} err="failed to get container status \"bd1306b94e6ee2c499edebf6519db2670c65e6d43545c8cd9191569bd66f5a84\": rpc error: code = NotFound desc = could not find container \"bd1306b94e6ee2c499edebf6519db2670c65e6d43545c8cd9191569bd66f5a84\": container with ID starting with bd1306b94e6ee2c499edebf6519db2670c65e6d43545c8cd9191569bd66f5a84 not found: ID does not exist" Nov 03 20:35:20.417343 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.417322 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g"] Nov 03 20:35:20.421199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:20.421161 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3aba31-6d4699b7b4-92x9g"] Nov 03 20:35:21.012636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.012538 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" containerID="cri-o://5d56270e330d2b3763f2a63e9b385deb8a2ae8c55b77949735acc0d77fc28523" gracePeriod=45 Nov 03 20:35:21.296542 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.296434 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="077732c1-4eda-4ce4-8afe-44e00e632df9" path="/var/lib/kubelet/pods/077732c1-4eda-4ce4-8afe-44e00e632df9/volumes" Nov 03 20:35:21.511197 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.510460 4918 generic.go:356] "Generic (PLEG): container finished" podID="fe6c2991-de3a-4374-a5c3-63816a171924" containerID="5d56270e330d2b3763f2a63e9b385deb8a2ae8c55b77949735acc0d77fc28523" exitCode=0 Nov 03 20:35:21.511197 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.510748 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" event={"ID":"fe6c2991-de3a-4374-a5c3-63816a171924","Type":"ContainerDied","Data":"5d56270e330d2b3763f2a63e9b385deb8a2ae8c55b77949735acc0d77fc28523"} Nov 03 20:35:21.511197 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.510804 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" event={"ID":"fe6c2991-de3a-4374-a5c3-63816a171924","Type":"ContainerDied","Data":"3cc6872f8bb7f85d204b1bcb009cb306b8942f2db3b92ebd2d8359fc17699130"} Nov 03 20:35:21.511197 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.510822 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cc6872f8bb7f85d204b1bcb009cb306b8942f2db3b92ebd2d8359fc17699130" Nov 03 20:35:21.550245 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.546381 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:35:21.623785 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.623740 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-dshm\") pod \"fe6c2991-de3a-4374-a5c3-63816a171924\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " Nov 03 20:35:21.647199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.629303 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe6c2991-de3a-4374-a5c3-63816a171924-kube-api-access-pv4mp\" (UniqueName: \"kubernetes.io/projected/fe6c2991-de3a-4374-a5c3-63816a171924-kube-api-access-pv4mp\") pod \"fe6c2991-de3a-4374-a5c3-63816a171924\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " Nov 03 20:35:21.647199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.629414 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-tmp-dir\") pod \"fe6c2991-de3a-4374-a5c3-63816a171924\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " Nov 03 20:35:21.647199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.629488 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-model-cache\") pod \"fe6c2991-de3a-4374-a5c3-63816a171924\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " Nov 03 20:35:21.647199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.629590 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-home\" (UniqueName: \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-home\") pod \"fe6c2991-de3a-4374-a5c3-63816a171924\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " Nov 03 20:35:21.647199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.629621 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe6c2991-de3a-4374-a5c3-63816a171924-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6c2991-de3a-4374-a5c3-63816a171924-tls-certs\") pod \"fe6c2991-de3a-4374-a5c3-63816a171924\" (UID: \"fe6c2991-de3a-4374-a5c3-63816a171924\") " Nov 03 20:35:21.647199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.631461 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-model-cache" pod "fe6c2991-de3a-4374-a5c3-63816a171924" (UID: "fe6c2991-de3a-4374-a5c3-63816a171924"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:21.647199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.631662 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-tmp-dir" pod "fe6c2991-de3a-4374-a5c3-63816a171924" (UID: "fe6c2991-de3a-4374-a5c3-63816a171924"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:21.647199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.639651 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-home" pod "fe6c2991-de3a-4374-a5c3-63816a171924" (UID: "fe6c2991-de3a-4374-a5c3-63816a171924"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:21.659967 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.659927 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe6c2991-de3a-4374-a5c3-63816a171924-tls-certs" pod "fe6c2991-de3a-4374-a5c3-63816a171924" (UID: "fe6c2991-de3a-4374-a5c3-63816a171924"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:21.663282 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.662972 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe6c2991-de3a-4374-a5c3-63816a171924-kube-api-access-pv4mp" pod "fe6c2991-de3a-4374-a5c3-63816a171924" (UID: "fe6c2991-de3a-4374-a5c3-63816a171924"). InnerVolumeSpecName "kube-api-access-pv4mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:21.667196 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.664290 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-dshm" pod "fe6c2991-de3a-4374-a5c3-63816a171924" (UID: "fe6c2991-de3a-4374-a5c3-63816a171924"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:21.731309 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.731108 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-dshm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.731309 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.731161 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv4mp\" (UniqueName: \"kubernetes.io/projected/fe6c2991-de3a-4374-a5c3-63816a171924-kube-api-access-pv4mp\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.731309 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.731197 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-tmp-dir\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.731309 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.731215 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-model-cache\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.731309 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.731229 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe6c2991-de3a-4374-a5c3-63816a171924-home\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.731309 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:21.731241 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6c2991-de3a-4374-a5c3-63816a171924-tls-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:22.557286 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:22.557216 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw" Nov 03 20:35:22.588164 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:22.588126 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw"] Nov 03 20:35:22.591020 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:22.590990 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/premium-w5-a6f98b-kserve-6c77fb66c7-h2vgw"] Nov 03 20:35:23.214495 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:23.214454 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" path="/var/lib/kubelet/pods/fe6c2991-de3a-4374-a5c3-63816a171924/volumes" Nov 03 20:35:26.340904 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:26.340129 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79"] Nov 03 20:35:26.340904 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:26.340409 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" podUID="4e08c21d-00c5-4769-87fc-253cb19a7c85" containerName="istio-proxy" containerID="cri-o://ab910217d99b91ab013369e3d6df3d63464b36cdb8c168fef4d1b11903e5bdcb" gracePeriod=30 Nov 03 20:35:28.064004 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:28.063968 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr"] Nov 03 20:35:28.064451 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:28.064201 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" podUID="645fc172-d7a8-48a6-93ec-6c9ed0fc0304" containerName="istio-proxy" containerID="cri-o://3402db34bde1d870cbd79c4673cff293024e825e9bf765a59df547e669031b8d" gracePeriod=30 Nov 03 20:35:28.244769 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:35:28.244726 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:35:29.065827 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.065792 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95"] Nov 03 20:35:29.270390 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.270355 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt"] Nov 03 20:35:29.270617 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.270590 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" podUID="d7646487-52b3-4fd9-8781-2d440af503a2" containerName="maas-api" containerID="cri-o://42f8d210a84f98baca99d18eb4c580ca90b3455b111588a53b31b2b91ae30b07" gracePeriod=30 Nov 03 20:35:29.567093 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.567066 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" Nov 03 20:35:29.601397 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.601361 4918 generic.go:356] "Generic (PLEG): container finished" podID="d7646487-52b3-4fd9-8781-2d440af503a2" containerID="42f8d210a84f98baca99d18eb4c580ca90b3455b111588a53b31b2b91ae30b07" exitCode=0 Nov 03 20:35:29.601558 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.601413 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" Nov 03 20:35:29.601558 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.601449 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" event={"ID":"d7646487-52b3-4fd9-8781-2d440af503a2","Type":"ContainerDied","Data":"42f8d210a84f98baca99d18eb4c580ca90b3455b111588a53b31b2b91ae30b07"} Nov 03 20:35:29.601558 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.601516 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt" event={"ID":"d7646487-52b3-4fd9-8781-2d440af503a2","Type":"ContainerDied","Data":"ccf0e34d0b8467878428196995e62cac4b5dcf6637a00d2f4d8648c2587ae065"} Nov 03 20:35:29.601558 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.601552 4918 scope.go:122] "RemoveContainer" containerID="42f8d210a84f98baca99d18eb4c580ca90b3455b111588a53b31b2b91ae30b07" Nov 03 20:35:29.615565 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.615541 4918 scope.go:122] "RemoveContainer" containerID="42f8d210a84f98baca99d18eb4c580ca90b3455b111588a53b31b2b91ae30b07" Nov 03 20:35:29.615994 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:35:29.615971 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42f8d210a84f98baca99d18eb4c580ca90b3455b111588a53b31b2b91ae30b07\": container with ID starting with 42f8d210a84f98baca99d18eb4c580ca90b3455b111588a53b31b2b91ae30b07 not found: ID does not exist" containerID="42f8d210a84f98baca99d18eb4c580ca90b3455b111588a53b31b2b91ae30b07" Nov 03 20:35:29.616126 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.616001 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42f8d210a84f98baca99d18eb4c580ca90b3455b111588a53b31b2b91ae30b07"} err="failed to get container status \"42f8d210a84f98baca99d18eb4c580ca90b3455b111588a53b31b2b91ae30b07\": rpc error: code = NotFound desc = could not find container \"42f8d210a84f98baca99d18eb4c580ca90b3455b111588a53b31b2b91ae30b07\": container with ID starting with 42f8d210a84f98baca99d18eb4c580ca90b3455b111588a53b31b2b91ae30b07 not found: ID does not exist" Nov 03 20:35:29.744720 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.744625 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7646487-52b3-4fd9-8781-2d440af503a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7646487-52b3-4fd9-8781-2d440af503a2-maas-api-tls\") pod \"d7646487-52b3-4fd9-8781-2d440af503a2\" (UID: \"d7646487-52b3-4fd9-8781-2d440af503a2\") " Nov 03 20:35:29.744891 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.744767 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7646487-52b3-4fd9-8781-2d440af503a2-kube-api-access-27lvt\" (UniqueName: \"kubernetes.io/projected/d7646487-52b3-4fd9-8781-2d440af503a2-kube-api-access-27lvt\") pod \"d7646487-52b3-4fd9-8781-2d440af503a2\" (UID: \"d7646487-52b3-4fd9-8781-2d440af503a2\") " Nov 03 20:35:29.744891 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.744801 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7646487-52b3-4fd9-8781-2d440af503a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7646487-52b3-4fd9-8781-2d440af503a2-metrics-certs\") pod \"d7646487-52b3-4fd9-8781-2d440af503a2\" (UID: \"d7646487-52b3-4fd9-8781-2d440af503a2\") " Nov 03 20:35:29.744891 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.744872 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7646487-52b3-4fd9-8781-2d440af503a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7646487-52b3-4fd9-8781-2d440af503a2-openshift-service-ca\") pod \"d7646487-52b3-4fd9-8781-2d440af503a2\" (UID: \"d7646487-52b3-4fd9-8781-2d440af503a2\") " Nov 03 20:35:29.745784 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.745754 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7646487-52b3-4fd9-8781-2d440af503a2-openshift-service-ca" pod "d7646487-52b3-4fd9-8781-2d440af503a2" (UID: "d7646487-52b3-4fd9-8781-2d440af503a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:29.753896 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.752999 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7646487-52b3-4fd9-8781-2d440af503a2-kube-api-access-27lvt" pod "d7646487-52b3-4fd9-8781-2d440af503a2" (UID: "d7646487-52b3-4fd9-8781-2d440af503a2"). InnerVolumeSpecName "kube-api-access-27lvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:29.753896 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.753043 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7646487-52b3-4fd9-8781-2d440af503a2-metrics-certs" pod "d7646487-52b3-4fd9-8781-2d440af503a2" (UID: "d7646487-52b3-4fd9-8781-2d440af503a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:29.757618 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.757589 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7646487-52b3-4fd9-8781-2d440af503a2-maas-api-tls" pod "d7646487-52b3-4fd9-8781-2d440af503a2" (UID: "d7646487-52b3-4fd9-8781-2d440af503a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:29.845972 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.845938 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7646487-52b3-4fd9-8781-2d440af503a2-openshift-service-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:29.845972 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.845969 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7646487-52b3-4fd9-8781-2d440af503a2-maas-api-tls\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:29.845972 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.845979 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27lvt\" (UniqueName: \"kubernetes.io/projected/d7646487-52b3-4fd9-8781-2d440af503a2-kube-api-access-27lvt\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:29.846228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.845988 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7646487-52b3-4fd9-8781-2d440af503a2-metrics-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:29.927674 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.927644 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt"] Nov 03 20:35:29.932402 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:29.932373 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-a6f98b-58949d6956-x2bqt"] Nov 03 20:35:31.214423 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.214388 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7646487-52b3-4fd9-8781-2d440af503a2" path="/var/lib/kubelet/pods/d7646487-52b3-4fd9-8781-2d440af503a2/volumes" Nov 03 20:35:31.606357 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.606331 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:35:31.613797 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.613767 4918 generic.go:356] "Generic (PLEG): container finished" podID="4e08c21d-00c5-4769-87fc-253cb19a7c85" containerID="ab910217d99b91ab013369e3d6df3d63464b36cdb8c168fef4d1b11903e5bdcb" exitCode=0 Nov 03 20:35:31.613926 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.613814 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" Nov 03 20:35:31.613926 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.613849 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" event={"ID":"4e08c21d-00c5-4769-87fc-253cb19a7c85","Type":"ContainerDied","Data":"ab910217d99b91ab013369e3d6df3d63464b36cdb8c168fef4d1b11903e5bdcb"} Nov 03 20:35:31.613926 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.613912 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79" event={"ID":"4e08c21d-00c5-4769-87fc-253cb19a7c85","Type":"ContainerDied","Data":"b71fe7813598caabf6cd458bc4b0c4db4222b53d8299766ae8ab5d0965a49429"} Nov 03 20:35:31.614099 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.613940 4918 scope.go:122] "RemoveContainer" containerID="ab910217d99b91ab013369e3d6df3d63464b36cdb8c168fef4d1b11903e5bdcb" Nov 03 20:35:31.623424 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.623402 4918 scope.go:122] "RemoveContainer" containerID="ab910217d99b91ab013369e3d6df3d63464b36cdb8c168fef4d1b11903e5bdcb" Nov 03 20:35:31.623712 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:35:31.623691 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab910217d99b91ab013369e3d6df3d63464b36cdb8c168fef4d1b11903e5bdcb\": container with ID starting with ab910217d99b91ab013369e3d6df3d63464b36cdb8c168fef4d1b11903e5bdcb not found: ID does not exist" containerID="ab910217d99b91ab013369e3d6df3d63464b36cdb8c168fef4d1b11903e5bdcb" Nov 03 20:35:31.623772 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.623722 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab910217d99b91ab013369e3d6df3d63464b36cdb8c168fef4d1b11903e5bdcb"} err="failed to get container status \"ab910217d99b91ab013369e3d6df3d63464b36cdb8c168fef4d1b11903e5bdcb\": rpc error: code = NotFound desc = could not find container \"ab910217d99b91ab013369e3d6df3d63464b36cdb8c168fef4d1b11903e5bdcb\": container with ID starting with ab910217d99b91ab013369e3d6df3d63464b36cdb8c168fef4d1b11903e5bdcb not found: ID does not exist" Nov 03 20:35:31.772326 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.772242 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e08c21d-00c5-4769-87fc-253cb19a7c85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e08c21d-00c5-4769-87fc-253cb19a7c85-istiod-ca-cert\") pod \"4e08c21d-00c5-4769-87fc-253cb19a7c85\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " Nov 03 20:35:31.772326 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.772299 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-envoy\") pod \"4e08c21d-00c5-4769-87fc-253cb19a7c85\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " Nov 03 20:35:31.772524 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.772390 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-credential-socket\") pod \"4e08c21d-00c5-4769-87fc-253cb19a7c85\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " Nov 03 20:35:31.772524 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.772443 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-workload-certs\") pod \"4e08c21d-00c5-4769-87fc-253cb19a7c85\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " Nov 03 20:35:31.772524 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.772468 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-workload-socket\") pod \"4e08c21d-00c5-4769-87fc-253cb19a7c85\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " Nov 03 20:35:31.772524 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.772511 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e08c21d-00c5-4769-87fc-253cb19a7c85-kube-api-access-h8rvn\" (UniqueName: \"kubernetes.io/projected/4e08c21d-00c5-4769-87fc-253cb19a7c85-kube-api-access-h8rvn\") pod \"4e08c21d-00c5-4769-87fc-253cb19a7c85\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " Nov 03 20:35:31.772742 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.772548 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-data\") pod \"4e08c21d-00c5-4769-87fc-253cb19a7c85\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " Nov 03 20:35:31.772742 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.772591 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-podinfo\") pod \"4e08c21d-00c5-4769-87fc-253cb19a7c85\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " Nov 03 20:35:31.772742 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.772660 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-token\" (UniqueName: \"kubernetes.io/projected/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-token\") pod \"4e08c21d-00c5-4769-87fc-253cb19a7c85\" (UID: \"4e08c21d-00c5-4769-87fc-253cb19a7c85\") " Nov 03 20:35:31.772742 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.772628 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e08c21d-00c5-4769-87fc-253cb19a7c85-istiod-ca-cert" pod "4e08c21d-00c5-4769-87fc-253cb19a7c85" (UID: "4e08c21d-00c5-4769-87fc-253cb19a7c85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:31.772938 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.772736 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-credential-socket" pod "4e08c21d-00c5-4769-87fc-253cb19a7c85" (UID: "4e08c21d-00c5-4769-87fc-253cb19a7c85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:31.772938 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.772764 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-workload-certs" pod "4e08c21d-00c5-4769-87fc-253cb19a7c85" (UID: "4e08c21d-00c5-4769-87fc-253cb19a7c85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:31.772938 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.772839 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-data" pod "4e08c21d-00c5-4769-87fc-253cb19a7c85" (UID: "4e08c21d-00c5-4769-87fc-253cb19a7c85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:31.772938 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.772928 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-workload-socket" pod "4e08c21d-00c5-4769-87fc-253cb19a7c85" (UID: "4e08c21d-00c5-4769-87fc-253cb19a7c85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:31.773155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.773136 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e08c21d-00c5-4769-87fc-253cb19a7c85-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.773252 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.773158 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.773252 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.773171 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.773252 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.773211 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.773252 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.773223 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.775131 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.775103 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-envoy" pod "4e08c21d-00c5-4769-87fc-253cb19a7c85" (UID: "4e08c21d-00c5-4769-87fc-253cb19a7c85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:31.775258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.775208 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-podinfo" pod "4e08c21d-00c5-4769-87fc-253cb19a7c85" (UID: "4e08c21d-00c5-4769-87fc-253cb19a7c85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:31.775420 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.775389 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e08c21d-00c5-4769-87fc-253cb19a7c85-kube-api-access-h8rvn" pod "4e08c21d-00c5-4769-87fc-253cb19a7c85" (UID: "4e08c21d-00c5-4769-87fc-253cb19a7c85"). InnerVolumeSpecName "kube-api-access-h8rvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:31.775514 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.775458 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-token" pod "4e08c21d-00c5-4769-87fc-253cb19a7c85" (UID: "4e08c21d-00c5-4769-87fc-253cb19a7c85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:31.874491 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.874455 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.874491 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.874487 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8rvn\" (UniqueName: \"kubernetes.io/projected/4e08c21d-00c5-4769-87fc-253cb19a7c85-kube-api-access-h8rvn\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.874491 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.874498 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.874712 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.874507 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e08c21d-00c5-4769-87fc-253cb19a7c85-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.938473 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.938441 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79"] Nov 03 20:35:31.942046 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:31.942019 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-2tr79"] Nov 03 20:35:33.215828 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.215789 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e08c21d-00c5-4769-87fc-253cb19a7c85" path="/var/lib/kubelet/pods/4e08c21d-00c5-4769-87fc-253cb19a7c85/volumes" Nov 03 20:35:33.328472 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.328443 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:35:33.391112 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.391022 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-kube-api-access-t7vpd\" (UniqueName: \"kubernetes.io/projected/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-kube-api-access-t7vpd\") pod \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " Nov 03 20:35:33.391300 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.391130 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-workload-certs\") pod \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " Nov 03 20:35:33.391300 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.391257 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-podinfo\") pod \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " Nov 03 20:35:33.391300 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.391291 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istiod-ca-cert\") pod \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " Nov 03 20:35:33.391505 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.391335 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-credential-socket\") pod \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " Nov 03 20:35:33.391505 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.391370 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-data\") pod \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " Nov 03 20:35:33.391505 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.391446 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-token\" (UniqueName: \"kubernetes.io/projected/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-token\") pod \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " Nov 03 20:35:33.391505 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.391479 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-workload-socket\") pod \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " Nov 03 20:35:33.391505 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.391490 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-workload-certs" pod "645fc172-d7a8-48a6-93ec-6c9ed0fc0304" (UID: "645fc172-d7a8-48a6-93ec-6c9ed0fc0304"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:33.391763 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.391556 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-envoy\") pod \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\" (UID: \"645fc172-d7a8-48a6-93ec-6c9ed0fc0304\") " Nov 03 20:35:33.391763 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.391716 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-workload-socket" pod "645fc172-d7a8-48a6-93ec-6c9ed0fc0304" (UID: "645fc172-d7a8-48a6-93ec-6c9ed0fc0304"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:33.392364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.391907 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-credential-socket" pod "645fc172-d7a8-48a6-93ec-6c9ed0fc0304" (UID: "645fc172-d7a8-48a6-93ec-6c9ed0fc0304"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:33.392364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.392163 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.392364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.392203 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.392364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.392217 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.392364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.391918 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-data" pod "645fc172-d7a8-48a6-93ec-6c9ed0fc0304" (UID: "645fc172-d7a8-48a6-93ec-6c9ed0fc0304"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:33.392688 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.392519 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istiod-ca-cert" pod "645fc172-d7a8-48a6-93ec-6c9ed0fc0304" (UID: "645fc172-d7a8-48a6-93ec-6c9ed0fc0304"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:33.394339 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.394148 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-kube-api-access-t7vpd" pod "645fc172-d7a8-48a6-93ec-6c9ed0fc0304" (UID: "645fc172-d7a8-48a6-93ec-6c9ed0fc0304"). InnerVolumeSpecName "kube-api-access-t7vpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:33.394339 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.394312 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-token" pod "645fc172-d7a8-48a6-93ec-6c9ed0fc0304" (UID: "645fc172-d7a8-48a6-93ec-6c9ed0fc0304"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:33.394587 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.394569 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-podinfo" pod "645fc172-d7a8-48a6-93ec-6c9ed0fc0304" (UID: "645fc172-d7a8-48a6-93ec-6c9ed0fc0304"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:33.395133 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.395111 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-envoy" pod "645fc172-d7a8-48a6-93ec-6c9ed0fc0304" (UID: "645fc172-d7a8-48a6-93ec-6c9ed0fc0304"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:33.493429 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.493392 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.493429 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.493425 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.493429 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.493435 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.493659 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.493447 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7vpd\" (UniqueName: \"kubernetes.io/projected/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-kube-api-access-t7vpd\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.493659 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.493464 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.493659 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.493472 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645fc172-d7a8-48a6-93ec-6c9ed0fc0304-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.627005 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.626970 4918 generic.go:356] "Generic (PLEG): container finished" podID="645fc172-d7a8-48a6-93ec-6c9ed0fc0304" containerID="3402db34bde1d870cbd79c4673cff293024e825e9bf765a59df547e669031b8d" exitCode=0 Nov 03 20:35:33.627240 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.627030 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" Nov 03 20:35:33.627240 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.627056 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" event={"ID":"645fc172-d7a8-48a6-93ec-6c9ed0fc0304","Type":"ContainerDied","Data":"3402db34bde1d870cbd79c4673cff293024e825e9bf765a59df547e669031b8d"} Nov 03 20:35:33.627240 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.627112 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr" event={"ID":"645fc172-d7a8-48a6-93ec-6c9ed0fc0304","Type":"ContainerDied","Data":"dee726375bc591efcedb045721d4ce49e39bce600d9ce06af4555f13cd4514fc"} Nov 03 20:35:33.627240 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.627135 4918 scope.go:122] "RemoveContainer" containerID="3402db34bde1d870cbd79c4673cff293024e825e9bf765a59df547e669031b8d" Nov 03 20:35:33.638582 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.638563 4918 scope.go:122] "RemoveContainer" containerID="07c212a00a781a6e576aa7189d906443fea4dbd04b72779b4a403a8180aecb35" Nov 03 20:35:33.647888 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.647872 4918 scope.go:122] "RemoveContainer" containerID="3402db34bde1d870cbd79c4673cff293024e825e9bf765a59df547e669031b8d" Nov 03 20:35:33.648279 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:35:33.648249 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3402db34bde1d870cbd79c4673cff293024e825e9bf765a59df547e669031b8d\": container with ID starting with 3402db34bde1d870cbd79c4673cff293024e825e9bf765a59df547e669031b8d not found: ID does not exist" containerID="3402db34bde1d870cbd79c4673cff293024e825e9bf765a59df547e669031b8d" Nov 03 20:35:33.648475 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.648292 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3402db34bde1d870cbd79c4673cff293024e825e9bf765a59df547e669031b8d"} err="failed to get container status \"3402db34bde1d870cbd79c4673cff293024e825e9bf765a59df547e669031b8d\": rpc error: code = NotFound desc = could not find container \"3402db34bde1d870cbd79c4673cff293024e825e9bf765a59df547e669031b8d\": container with ID starting with 3402db34bde1d870cbd79c4673cff293024e825e9bf765a59df547e669031b8d not found: ID does not exist" Nov 03 20:35:33.648475 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.648317 4918 scope.go:122] "RemoveContainer" containerID="07c212a00a781a6e576aa7189d906443fea4dbd04b72779b4a403a8180aecb35" Nov 03 20:35:33.648762 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:35:33.648722 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c212a00a781a6e576aa7189d906443fea4dbd04b72779b4a403a8180aecb35\": container with ID starting with 07c212a00a781a6e576aa7189d906443fea4dbd04b72779b4a403a8180aecb35 not found: ID does not exist" containerID="07c212a00a781a6e576aa7189d906443fea4dbd04b72779b4a403a8180aecb35" Nov 03 20:35:33.648859 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.648756 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c212a00a781a6e576aa7189d906443fea4dbd04b72779b4a403a8180aecb35"} err="failed to get container status \"07c212a00a781a6e576aa7189d906443fea4dbd04b72779b4a403a8180aecb35\": rpc error: code = NotFound desc = could not find container \"07c212a00a781a6e576aa7189d906443fea4dbd04b72779b4a403a8180aecb35\": container with ID starting with 07c212a00a781a6e576aa7189d906443fea4dbd04b72779b4a403a8180aecb35 not found: ID does not exist" Nov 03 20:35:33.651364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.651344 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr"] Nov 03 20:35:33.659959 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.658936 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-svzhr"] Nov 03 20:35:33.665440 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.665408 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" podUID="4aaf5a9a-91e9-48d8-b69d-97de7d99d51a" containerName="main" containerID="cri-o://ddded8e2ea8ab7846d5a6475e41b70207e050debee74c55a0faa1b0ec9f66f73" gracePeriod=45 Nov 03 20:35:33.917537 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.917509 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:35:33.997771 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.997740 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-tmp-dir\") pod \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " Nov 03 20:35:33.997965 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.997781 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-kube-api-access-sztw7\" (UniqueName: \"kubernetes.io/projected/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-kube-api-access-sztw7\") pod \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " Nov 03 20:35:33.997965 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.997826 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-tls-certs\" (UniqueName: \"kubernetes.io/secret/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-tls-certs\") pod \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " Nov 03 20:35:33.997965 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.997846 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-home\" (UniqueName: \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-home\") pod \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " Nov 03 20:35:33.997965 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.997896 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-dshm\") pod \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " Nov 03 20:35:33.997965 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.997956 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-model-cache\") pod \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\" (UID: \"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a\") " Nov 03 20:35:33.998257 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.998092 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-tmp-dir" pod "4aaf5a9a-91e9-48d8-b69d-97de7d99d51a" (UID: "4aaf5a9a-91e9-48d8-b69d-97de7d99d51a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:33.998257 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.998136 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-home" pod "4aaf5a9a-91e9-48d8-b69d-97de7d99d51a" (UID: "4aaf5a9a-91e9-48d8-b69d-97de7d99d51a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:33.998361 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.998261 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-model-cache" pod "4aaf5a9a-91e9-48d8-b69d-97de7d99d51a" (UID: "4aaf5a9a-91e9-48d8-b69d-97de7d99d51a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:33.998460 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.998440 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-tmp-dir\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.998502 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.998466 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-home\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.998502 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:33.998479 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-model-cache\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:34.000234 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:34.000209 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-tls-certs" pod "4aaf5a9a-91e9-48d8-b69d-97de7d99d51a" (UID: "4aaf5a9a-91e9-48d8-b69d-97de7d99d51a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:34.000234 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:34.000224 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-dshm" pod "4aaf5a9a-91e9-48d8-b69d-97de7d99d51a" (UID: "4aaf5a9a-91e9-48d8-b69d-97de7d99d51a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:34.000354 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:34.000271 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-kube-api-access-sztw7" pod "4aaf5a9a-91e9-48d8-b69d-97de7d99d51a" (UID: "4aaf5a9a-91e9-48d8-b69d-97de7d99d51a"). InnerVolumeSpecName "kube-api-access-sztw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:34.066441 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:34.066400 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95" podUID="be8a9e79-44b3-4610-acbc-ee4dd69c121f" containerName="payload-processing" containerID="cri-o://abc0e897275b5d77f1eaffa2d86821926b2fd783b2a911fcf82a09c4a00df8ab" gracePeriod=25 Nov 03 20:35:34.099635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:34.099606 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sztw7\" (UniqueName: \"kubernetes.io/projected/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-kube-api-access-sztw7\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:34.099635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:34.099634 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-tls-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:34.099635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:34.099645 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a-dshm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:34.634112 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:34.634075 4918 generic.go:356] "Generic (PLEG): container finished" podID="4aaf5a9a-91e9-48d8-b69d-97de7d99d51a" containerID="ddded8e2ea8ab7846d5a6475e41b70207e050debee74c55a0faa1b0ec9f66f73" exitCode=0 Nov 03 20:35:34.634546 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:34.634124 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" Nov 03 20:35:34.634546 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:34.634155 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" event={"ID":"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a","Type":"ContainerDied","Data":"ddded8e2ea8ab7846d5a6475e41b70207e050debee74c55a0faa1b0ec9f66f73"} Nov 03 20:35:34.634546 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:34.634216 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m" event={"ID":"4aaf5a9a-91e9-48d8-b69d-97de7d99d51a","Type":"ContainerDied","Data":"627b0a8206bd36cb6c86c73d4de06e7daffd64c494df0217f26e8afd6f3462d1"} Nov 03 20:35:34.634546 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:34.634241 4918 scope.go:122] "RemoveContainer" containerID="ddded8e2ea8ab7846d5a6475e41b70207e050debee74c55a0faa1b0ec9f66f73" Nov 03 20:35:34.643475 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:34.643432 4918 scope.go:122] "RemoveContainer" containerID="ddded8e2ea8ab7846d5a6475e41b70207e050debee74c55a0faa1b0ec9f66f73" Nov 03 20:35:34.643722 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:35:34.643705 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddded8e2ea8ab7846d5a6475e41b70207e050debee74c55a0faa1b0ec9f66f73\": container with ID starting with ddded8e2ea8ab7846d5a6475e41b70207e050debee74c55a0faa1b0ec9f66f73 not found: ID does not exist" containerID="ddded8e2ea8ab7846d5a6475e41b70207e050debee74c55a0faa1b0ec9f66f73" Nov 03 20:35:34.643789 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:34.643727 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddded8e2ea8ab7846d5a6475e41b70207e050debee74c55a0faa1b0ec9f66f73"} err="failed to get container status \"ddded8e2ea8ab7846d5a6475e41b70207e050debee74c55a0faa1b0ec9f66f73\": rpc error: code = NotFound desc = could not find container \"ddded8e2ea8ab7846d5a6475e41b70207e050debee74c55a0faa1b0ec9f66f73\": container with ID starting with ddded8e2ea8ab7846d5a6475e41b70207e050debee74c55a0faa1b0ec9f66f73 not found: ID does not exist" Nov 03 20:35:34.657305 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:34.657275 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m"] Nov 03 20:35:34.660983 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:34.660958 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c60d2a/test-model-c60d2a-kserve-65d5f4b98c-pc68m"] Nov 03 20:35:35.214054 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:35.214019 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aaf5a9a-91e9-48d8-b69d-97de7d99d51a" path="/var/lib/kubelet/pods/4aaf5a9a-91e9-48d8-b69d-97de7d99d51a/volumes" Nov 03 20:35:35.214537 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:35.214521 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="645fc172-d7a8-48a6-93ec-6c9ed0fc0304" path="/var/lib/kubelet/pods/645fc172-d7a8-48a6-93ec-6c9ed0fc0304/volumes" Nov 03 20:35:36.297391 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:36.297348 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95" podUID="be8a9e79-44b3-4610-acbc-ee4dd69c121f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:35:38.922914 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:38.922877 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j"] Nov 03 20:35:38.923469 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:38.923162 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" podUID="b92265eb-94c4-42c6-b2c0-b22feab2eb07" containerName="istio-proxy" containerID="cri-o://3d8d88500ae9dedf995428040759058266fc3fbad157e04e029e80de2195f10a" gracePeriod=30 Nov 03 20:35:41.791483 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:41.791445 4918 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:35:41.791885 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:41.791509 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" podUID="b92265eb-94c4-42c6-b2c0-b22feab2eb07" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:35:44.194632 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.194568 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:35:44.321418 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.321373 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istiod-ca-cert\") pod \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " Nov 03 20:35:44.321617 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.321463 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-envoy\") pod \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " Nov 03 20:35:44.321617 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.321529 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-token\" (UniqueName: \"kubernetes.io/projected/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-token\") pod \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " Nov 03 20:35:44.321617 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.321552 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-podinfo\") pod \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " Nov 03 20:35:44.321617 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.321575 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-workload-certs\") pod \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " Nov 03 20:35:44.321863 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.321638 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92265eb-94c4-42c6-b2c0-b22feab2eb07-kube-api-access-rc569\" (UniqueName: \"kubernetes.io/projected/b92265eb-94c4-42c6-b2c0-b22feab2eb07-kube-api-access-rc569\") pod \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " Nov 03 20:35:44.321863 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.321677 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-data\") pod \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " Nov 03 20:35:44.321863 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.321698 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-workload-socket\") pod \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " Nov 03 20:35:44.321863 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.321740 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-credential-socket\") pod \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\" (UID: \"b92265eb-94c4-42c6-b2c0-b22feab2eb07\") " Nov 03 20:35:44.321863 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.321818 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istiod-ca-cert" pod "b92265eb-94c4-42c6-b2c0-b22feab2eb07" (UID: "b92265eb-94c4-42c6-b2c0-b22feab2eb07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:44.322130 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.321956 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-workload-certs" pod "b92265eb-94c4-42c6-b2c0-b22feab2eb07" (UID: "b92265eb-94c4-42c6-b2c0-b22feab2eb07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:44.322130 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.321982 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-data" pod "b92265eb-94c4-42c6-b2c0-b22feab2eb07" (UID: "b92265eb-94c4-42c6-b2c0-b22feab2eb07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:44.322264 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.322211 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-workload-socket" pod "b92265eb-94c4-42c6-b2c0-b22feab2eb07" (UID: "b92265eb-94c4-42c6-b2c0-b22feab2eb07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:44.322372 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.322351 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.322441 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.322376 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.322441 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.322383 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-credential-socket" pod "b92265eb-94c4-42c6-b2c0-b22feab2eb07" (UID: "b92265eb-94c4-42c6-b2c0-b22feab2eb07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:44.322441 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.322391 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.322441 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.322426 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.323889 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.323859 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-podinfo" pod "b92265eb-94c4-42c6-b2c0-b22feab2eb07" (UID: "b92265eb-94c4-42c6-b2c0-b22feab2eb07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:44.324017 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.323898 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-token" pod "b92265eb-94c4-42c6-b2c0-b22feab2eb07" (UID: "b92265eb-94c4-42c6-b2c0-b22feab2eb07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:44.324017 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.323960 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-envoy" pod "b92265eb-94c4-42c6-b2c0-b22feab2eb07" (UID: "b92265eb-94c4-42c6-b2c0-b22feab2eb07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:44.324098 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.324061 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92265eb-94c4-42c6-b2c0-b22feab2eb07-kube-api-access-rc569" pod "b92265eb-94c4-42c6-b2c0-b22feab2eb07" (UID: "b92265eb-94c4-42c6-b2c0-b22feab2eb07"). InnerVolumeSpecName "kube-api-access-rc569". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:44.423344 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.423305 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.423344 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.423337 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.423344 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.423350 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92265eb-94c4-42c6-b2c0-b22feab2eb07-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.423597 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.423359 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc569\" (UniqueName: \"kubernetes.io/projected/b92265eb-94c4-42c6-b2c0-b22feab2eb07-kube-api-access-rc569\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.423597 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.423370 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92265eb-94c4-42c6-b2c0-b22feab2eb07-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.695172 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.695137 4918 generic.go:356] "Generic (PLEG): container finished" podID="b92265eb-94c4-42c6-b2c0-b22feab2eb07" containerID="3d8d88500ae9dedf995428040759058266fc3fbad157e04e029e80de2195f10a" exitCode=0 Nov 03 20:35:44.695362 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.695218 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" Nov 03 20:35:44.695362 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.695210 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" event={"ID":"b92265eb-94c4-42c6-b2c0-b22feab2eb07","Type":"ContainerDied","Data":"3d8d88500ae9dedf995428040759058266fc3fbad157e04e029e80de2195f10a"} Nov 03 20:35:44.695362 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.695357 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j" event={"ID":"b92265eb-94c4-42c6-b2c0-b22feab2eb07","Type":"ContainerDied","Data":"38f117737cfe7b1c75d7c91f4c8d9a6c18312a338d35ca21ebb7d27251842972"} Nov 03 20:35:44.695477 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.695382 4918 scope.go:122] "RemoveContainer" containerID="3d8d88500ae9dedf995428040759058266fc3fbad157e04e029e80de2195f10a" Nov 03 20:35:44.706065 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.706029 4918 scope.go:122] "RemoveContainer" containerID="91477df13a4f8468eabf40a759fe67b4c0a795667559c47f983c329aec919025" Nov 03 20:35:44.714820 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.714800 4918 scope.go:122] "RemoveContainer" containerID="3d8d88500ae9dedf995428040759058266fc3fbad157e04e029e80de2195f10a" Nov 03 20:35:44.715083 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:35:44.715064 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d8d88500ae9dedf995428040759058266fc3fbad157e04e029e80de2195f10a\": container with ID starting with 3d8d88500ae9dedf995428040759058266fc3fbad157e04e029e80de2195f10a not found: ID does not exist" containerID="3d8d88500ae9dedf995428040759058266fc3fbad157e04e029e80de2195f10a" Nov 03 20:35:44.715165 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.715096 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d8d88500ae9dedf995428040759058266fc3fbad157e04e029e80de2195f10a"} err="failed to get container status \"3d8d88500ae9dedf995428040759058266fc3fbad157e04e029e80de2195f10a\": rpc error: code = NotFound desc = could not find container \"3d8d88500ae9dedf995428040759058266fc3fbad157e04e029e80de2195f10a\": container with ID starting with 3d8d88500ae9dedf995428040759058266fc3fbad157e04e029e80de2195f10a not found: ID does not exist" Nov 03 20:35:44.715165 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.715120 4918 scope.go:122] "RemoveContainer" containerID="91477df13a4f8468eabf40a759fe67b4c0a795667559c47f983c329aec919025" Nov 03 20:35:44.715387 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:35:44.715370 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91477df13a4f8468eabf40a759fe67b4c0a795667559c47f983c329aec919025\": container with ID starting with 91477df13a4f8468eabf40a759fe67b4c0a795667559c47f983c329aec919025 not found: ID does not exist" containerID="91477df13a4f8468eabf40a759fe67b4c0a795667559c47f983c329aec919025" Nov 03 20:35:44.715442 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.715396 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91477df13a4f8468eabf40a759fe67b4c0a795667559c47f983c329aec919025"} err="failed to get container status \"91477df13a4f8468eabf40a759fe67b4c0a795667559c47f983c329aec919025\": rpc error: code = NotFound desc = could not find container \"91477df13a4f8468eabf40a759fe67b4c0a795667559c47f983c329aec919025\": container with ID starting with 91477df13a4f8468eabf40a759fe67b4c0a795667559c47f983c329aec919025 not found: ID does not exist" Nov 03 20:35:44.718607 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.718585 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j"] Nov 03 20:35:44.724699 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:44.724677 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-jf26j"] Nov 03 20:35:45.214083 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:45.214042 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b92265eb-94c4-42c6-b2c0-b22feab2eb07" path="/var/lib/kubelet/pods/b92265eb-94c4-42c6-b2c0-b22feab2eb07/volumes" Nov 03 20:35:46.296780 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:46.296742 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95" podUID="be8a9e79-44b3-4610-acbc-ee4dd69c121f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:35:47.126411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:47.126374 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf"] Nov 03 20:35:47.126820 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:47.126747 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" podUID="22993cbd-6df5-493a-b19a-3f1269dbc57c" containerName="istio-proxy" containerID="cri-o://37ddbaa9ecad3ef24ff76a2ea1b88848a94ccd7c198e440a96376cb5950e6c8e" gracePeriod=30 Nov 03 20:35:52.392100 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.392040 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:35:52.504677 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.504643 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-token\" (UniqueName: \"kubernetes.io/projected/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-token\") pod \"22993cbd-6df5-493a-b19a-3f1269dbc57c\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " Nov 03 20:35:52.504867 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.504687 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-workload-certs\") pod \"22993cbd-6df5-493a-b19a-3f1269dbc57c\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " Nov 03 20:35:52.504867 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.504723 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-data\") pod \"22993cbd-6df5-493a-b19a-3f1269dbc57c\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " Nov 03 20:35:52.504867 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.504741 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-credential-socket\") pod \"22993cbd-6df5-493a-b19a-3f1269dbc57c\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " Nov 03 20:35:52.504867 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.504800 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-podinfo\") pod \"22993cbd-6df5-493a-b19a-3f1269dbc57c\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " Nov 03 20:35:52.504867 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.504828 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22993cbd-6df5-493a-b19a-3f1269dbc57c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22993cbd-6df5-493a-b19a-3f1269dbc57c-istiod-ca-cert\") pod \"22993cbd-6df5-493a-b19a-3f1269dbc57c\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " Nov 03 20:35:52.505135 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.504936 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22993cbd-6df5-493a-b19a-3f1269dbc57c-kube-api-access-qzsxf\" (UniqueName: \"kubernetes.io/projected/22993cbd-6df5-493a-b19a-3f1269dbc57c-kube-api-access-qzsxf\") pod \"22993cbd-6df5-493a-b19a-3f1269dbc57c\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " Nov 03 20:35:52.505135 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.504970 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-workload-socket\") pod \"22993cbd-6df5-493a-b19a-3f1269dbc57c\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " Nov 03 20:35:52.505135 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.505022 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-envoy\") pod \"22993cbd-6df5-493a-b19a-3f1269dbc57c\" (UID: \"22993cbd-6df5-493a-b19a-3f1269dbc57c\") " Nov 03 20:35:52.505135 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.505063 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-data" pod "22993cbd-6df5-493a-b19a-3f1269dbc57c" (UID: "22993cbd-6df5-493a-b19a-3f1269dbc57c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:52.505135 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.505124 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-credential-socket" pod "22993cbd-6df5-493a-b19a-3f1269dbc57c" (UID: "22993cbd-6df5-493a-b19a-3f1269dbc57c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:52.505428 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.505286 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22993cbd-6df5-493a-b19a-3f1269dbc57c-istiod-ca-cert" pod "22993cbd-6df5-493a-b19a-3f1269dbc57c" (UID: "22993cbd-6df5-493a-b19a-3f1269dbc57c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:52.505428 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.505371 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-workload-socket" pod "22993cbd-6df5-493a-b19a-3f1269dbc57c" (UID: "22993cbd-6df5-493a-b19a-3f1269dbc57c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:52.505428 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.505400 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-workload-certs" pod "22993cbd-6df5-493a-b19a-3f1269dbc57c" (UID: "22993cbd-6df5-493a-b19a-3f1269dbc57c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:52.505607 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.505591 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22993cbd-6df5-493a-b19a-3f1269dbc57c-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.505668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.505615 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.505668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.505626 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.505668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.505640 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.505668 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.505655 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.506985 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.506953 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-token" pod "22993cbd-6df5-493a-b19a-3f1269dbc57c" (UID: "22993cbd-6df5-493a-b19a-3f1269dbc57c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:52.507299 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.507280 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-envoy" pod "22993cbd-6df5-493a-b19a-3f1269dbc57c" (UID: "22993cbd-6df5-493a-b19a-3f1269dbc57c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:52.507523 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.507500 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22993cbd-6df5-493a-b19a-3f1269dbc57c-kube-api-access-qzsxf" pod "22993cbd-6df5-493a-b19a-3f1269dbc57c" (UID: "22993cbd-6df5-493a-b19a-3f1269dbc57c"). InnerVolumeSpecName "kube-api-access-qzsxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:52.507523 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.507511 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-podinfo" pod "22993cbd-6df5-493a-b19a-3f1269dbc57c" (UID: "22993cbd-6df5-493a-b19a-3f1269dbc57c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:52.606834 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.606792 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.606834 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.606831 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzsxf\" (UniqueName: \"kubernetes.io/projected/22993cbd-6df5-493a-b19a-3f1269dbc57c-kube-api-access-qzsxf\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.607044 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.606847 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.607044 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.606860 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22993cbd-6df5-493a-b19a-3f1269dbc57c-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.747209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.747112 4918 generic.go:356] "Generic (PLEG): container finished" podID="22993cbd-6df5-493a-b19a-3f1269dbc57c" containerID="37ddbaa9ecad3ef24ff76a2ea1b88848a94ccd7c198e440a96376cb5950e6c8e" exitCode=0 Nov 03 20:35:52.747209 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.747166 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" Nov 03 20:35:52.747453 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.747211 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" event={"ID":"22993cbd-6df5-493a-b19a-3f1269dbc57c","Type":"ContainerDied","Data":"37ddbaa9ecad3ef24ff76a2ea1b88848a94ccd7c198e440a96376cb5950e6c8e"} Nov 03 20:35:52.747453 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.747281 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf" event={"ID":"22993cbd-6df5-493a-b19a-3f1269dbc57c","Type":"ContainerDied","Data":"271b1622425b4ced6863d6583ecc6de4b8917258929a56a35ca99f5c670599df"} Nov 03 20:35:52.747453 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.747311 4918 scope.go:122] "RemoveContainer" containerID="37ddbaa9ecad3ef24ff76a2ea1b88848a94ccd7c198e440a96376cb5950e6c8e" Nov 03 20:35:52.758158 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.758142 4918 scope.go:122] "RemoveContainer" containerID="37ddbaa9ecad3ef24ff76a2ea1b88848a94ccd7c198e440a96376cb5950e6c8e" Nov 03 20:35:52.758472 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:35:52.758453 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ddbaa9ecad3ef24ff76a2ea1b88848a94ccd7c198e440a96376cb5950e6c8e\": container with ID starting with 37ddbaa9ecad3ef24ff76a2ea1b88848a94ccd7c198e440a96376cb5950e6c8e not found: ID does not exist" containerID="37ddbaa9ecad3ef24ff76a2ea1b88848a94ccd7c198e440a96376cb5950e6c8e" Nov 03 20:35:52.758520 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.758483 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ddbaa9ecad3ef24ff76a2ea1b88848a94ccd7c198e440a96376cb5950e6c8e"} err="failed to get container status \"37ddbaa9ecad3ef24ff76a2ea1b88848a94ccd7c198e440a96376cb5950e6c8e\": rpc error: code = NotFound desc = could not find container \"37ddbaa9ecad3ef24ff76a2ea1b88848a94ccd7c198e440a96376cb5950e6c8e\": container with ID starting with 37ddbaa9ecad3ef24ff76a2ea1b88848a94ccd7c198e440a96376cb5950e6c8e not found: ID does not exist" Nov 03 20:35:52.775917 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.775892 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf"] Nov 03 20:35:52.780017 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:52.779995 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-c7xvf"] Nov 03 20:35:53.214412 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:53.214368 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22993cbd-6df5-493a-b19a-3f1269dbc57c" path="/var/lib/kubelet/pods/22993cbd-6df5-493a-b19a-3f1269dbc57c/volumes" Nov 03 20:35:54.325658 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:54.325599 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95" Nov 03 20:35:54.426614 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:54.426585 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be8a9e79-44b3-4610-acbc-ee4dd69c121f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be8a9e79-44b3-4610-acbc-ee4dd69c121f-plugins-config-volume\") pod \"be8a9e79-44b3-4610-acbc-ee4dd69c121f\" (UID: \"be8a9e79-44b3-4610-acbc-ee4dd69c121f\") " Nov 03 20:35:54.426801 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:54.426639 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be8a9e79-44b3-4610-acbc-ee4dd69c121f-kube-api-access-zssff\" (UniqueName: \"kubernetes.io/projected/be8a9e79-44b3-4610-acbc-ee4dd69c121f-kube-api-access-zssff\") pod \"be8a9e79-44b3-4610-acbc-ee4dd69c121f\" (UID: \"be8a9e79-44b3-4610-acbc-ee4dd69c121f\") " Nov 03 20:35:54.427053 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:54.427029 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8a9e79-44b3-4610-acbc-ee4dd69c121f-plugins-config-volume" pod "be8a9e79-44b3-4610-acbc-ee4dd69c121f" (UID: "be8a9e79-44b3-4610-acbc-ee4dd69c121f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:54.427144 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:54.427129 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be8a9e79-44b3-4610-acbc-ee4dd69c121f-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:54.428787 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:54.428760 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8a9e79-44b3-4610-acbc-ee4dd69c121f-kube-api-access-zssff" pod "be8a9e79-44b3-4610-acbc-ee4dd69c121f" (UID: "be8a9e79-44b3-4610-acbc-ee4dd69c121f"). InnerVolumeSpecName "kube-api-access-zssff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:54.528270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:54.528241 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zssff\" (UniqueName: \"kubernetes.io/projected/be8a9e79-44b3-4610-acbc-ee4dd69c121f-kube-api-access-zssff\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:54.761523 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:54.761435 4918 generic.go:356] "Generic (PLEG): container finished" podID="be8a9e79-44b3-4610-acbc-ee4dd69c121f" containerID="abc0e897275b5d77f1eaffa2d86821926b2fd783b2a911fcf82a09c4a00df8ab" exitCode=0 Nov 03 20:35:54.761523 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:54.761492 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95" Nov 03 20:35:54.761705 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:54.761496 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95" event={"ID":"be8a9e79-44b3-4610-acbc-ee4dd69c121f","Type":"ContainerDied","Data":"abc0e897275b5d77f1eaffa2d86821926b2fd783b2a911fcf82a09c4a00df8ab"} Nov 03 20:35:54.761705 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:54.761614 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95" event={"ID":"be8a9e79-44b3-4610-acbc-ee4dd69c121f","Type":"ContainerDied","Data":"fd796ce2e12ed7c57c09ea95b2d6556483dcd58c1ba3254652c2d8be3e66de90"} Nov 03 20:35:54.761705 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:54.761642 4918 scope.go:122] "RemoveContainer" containerID="abc0e897275b5d77f1eaffa2d86821926b2fd783b2a911fcf82a09c4a00df8ab" Nov 03 20:35:54.771834 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:54.771814 4918 scope.go:122] "RemoveContainer" containerID="abc0e897275b5d77f1eaffa2d86821926b2fd783b2a911fcf82a09c4a00df8ab" Nov 03 20:35:54.772102 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:35:54.772080 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abc0e897275b5d77f1eaffa2d86821926b2fd783b2a911fcf82a09c4a00df8ab\": container with ID starting with abc0e897275b5d77f1eaffa2d86821926b2fd783b2a911fcf82a09c4a00df8ab not found: ID does not exist" containerID="abc0e897275b5d77f1eaffa2d86821926b2fd783b2a911fcf82a09c4a00df8ab" Nov 03 20:35:54.772152 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:54.772110 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abc0e897275b5d77f1eaffa2d86821926b2fd783b2a911fcf82a09c4a00df8ab"} err="failed to get container status \"abc0e897275b5d77f1eaffa2d86821926b2fd783b2a911fcf82a09c4a00df8ab\": rpc error: code = NotFound desc = could not find container \"abc0e897275b5d77f1eaffa2d86821926b2fd783b2a911fcf82a09c4a00df8ab\": container with ID starting with abc0e897275b5d77f1eaffa2d86821926b2fd783b2a911fcf82a09c4a00df8ab not found: ID does not exist" Nov 03 20:35:54.784932 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:54.784907 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95"] Nov 03 20:35:54.790108 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:54.790084 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-45e569-5f695b9f48-hkq95"] Nov 03 20:35:55.222576 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:55.222539 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be8a9e79-44b3-4610-acbc-ee4dd69c121f" path="/var/lib/kubelet/pods/be8a9e79-44b3-4610-acbc-ee4dd69c121f/volumes" Nov 03 20:35:58.207250 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:35:58.207208 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:35:59.929952 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:35:59.929918 4918 scope.go:122] "RemoveContainer" containerID="5d56270e330d2b3763f2a63e9b385deb8a2ae8c55b77949735acc0d77fc28523" Nov 03 20:36:05.764060 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:05.763756 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4"] Nov 03 20:36:05.764060 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:05.764007 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" podUID="942fc70c-abe0-4988-b6f9-c9babff54817" containerName="istio-proxy" containerID="cri-o://b040b847214a2009d427e3ea93b4bd146323adf4602fb0033ea9aab279e6e353" gracePeriod=30 Nov 03 20:36:11.023600 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.023572 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:36:11.110636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.110604 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/942fc70c-abe0-4988-b6f9-c9babff54817-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942fc70c-abe0-4988-b6f9-c9babff54817-istio-podinfo\") pod \"942fc70c-abe0-4988-b6f9-c9babff54817\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " Nov 03 20:36:11.110841 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.110660 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-workload-certs\") pod \"942fc70c-abe0-4988-b6f9-c9babff54817\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " Nov 03 20:36:11.110841 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.110705 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-istio-data\") pod \"942fc70c-abe0-4988-b6f9-c9babff54817\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " Nov 03 20:36:11.110841 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.110734 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-istio-envoy\") pod \"942fc70c-abe0-4988-b6f9-c9babff54817\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " Nov 03 20:36:11.110841 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.110772 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/942fc70c-abe0-4988-b6f9-c9babff54817-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942fc70c-abe0-4988-b6f9-c9babff54817-istiod-ca-cert\") pod \"942fc70c-abe0-4988-b6f9-c9babff54817\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " Nov 03 20:36:11.111082 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.110890 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-workload-socket\") pod \"942fc70c-abe0-4988-b6f9-c9babff54817\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " Nov 03 20:36:11.111082 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.110954 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-workload-certs" pod "942fc70c-abe0-4988-b6f9-c9babff54817" (UID: "942fc70c-abe0-4988-b6f9-c9babff54817"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:11.111082 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.110963 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942fc70c-abe0-4988-b6f9-c9babff54817-istio-token\" (UniqueName: \"kubernetes.io/projected/942fc70c-abe0-4988-b6f9-c9babff54817-istio-token\") pod \"942fc70c-abe0-4988-b6f9-c9babff54817\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " Nov 03 20:36:11.111082 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.111056 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-credential-socket\") pod \"942fc70c-abe0-4988-b6f9-c9babff54817\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " Nov 03 20:36:11.111082 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.111062 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-istio-data" pod "942fc70c-abe0-4988-b6f9-c9babff54817" (UID: "942fc70c-abe0-4988-b6f9-c9babff54817"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:11.111361 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.111157 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942fc70c-abe0-4988-b6f9-c9babff54817-kube-api-access-n7zxn\" (UniqueName: \"kubernetes.io/projected/942fc70c-abe0-4988-b6f9-c9babff54817-kube-api-access-n7zxn\") pod \"942fc70c-abe0-4988-b6f9-c9babff54817\" (UID: \"942fc70c-abe0-4988-b6f9-c9babff54817\") " Nov 03 20:36:11.111361 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.111222 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-workload-socket" pod "942fc70c-abe0-4988-b6f9-c9babff54817" (UID: "942fc70c-abe0-4988-b6f9-c9babff54817"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:11.111361 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.111274 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-credential-socket" pod "942fc70c-abe0-4988-b6f9-c9babff54817" (UID: "942fc70c-abe0-4988-b6f9-c9babff54817"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:11.111361 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.111321 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942fc70c-abe0-4988-b6f9-c9babff54817-istiod-ca-cert" pod "942fc70c-abe0-4988-b6f9-c9babff54817" (UID: "942fc70c-abe0-4988-b6f9-c9babff54817"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:11.111607 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.111583 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:11.111681 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.111612 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:11.111681 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.111627 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:11.111681 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.111640 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:11.111681 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.111653 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942fc70c-abe0-4988-b6f9-c9babff54817-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:11.113026 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.113000 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-istio-envoy" pod "942fc70c-abe0-4988-b6f9-c9babff54817" (UID: "942fc70c-abe0-4988-b6f9-c9babff54817"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:11.113202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.113156 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942fc70c-abe0-4988-b6f9-c9babff54817-kube-api-access-n7zxn" pod "942fc70c-abe0-4988-b6f9-c9babff54817" (UID: "942fc70c-abe0-4988-b6f9-c9babff54817"). InnerVolumeSpecName "kube-api-access-n7zxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:11.113279 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.113243 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/942fc70c-abe0-4988-b6f9-c9babff54817-istio-podinfo" pod "942fc70c-abe0-4988-b6f9-c9babff54817" (UID: "942fc70c-abe0-4988-b6f9-c9babff54817"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:36:11.113340 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.113321 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942fc70c-abe0-4988-b6f9-c9babff54817-istio-token" pod "942fc70c-abe0-4988-b6f9-c9babff54817" (UID: "942fc70c-abe0-4988-b6f9-c9babff54817"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:11.212579 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.212535 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942fc70c-abe0-4988-b6f9-c9babff54817-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:11.212579 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.212567 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942fc70c-abe0-4988-b6f9-c9babff54817-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:11.212579 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.212577 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7zxn\" (UniqueName: \"kubernetes.io/projected/942fc70c-abe0-4988-b6f9-c9babff54817-kube-api-access-n7zxn\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:11.212579 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.212588 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942fc70c-abe0-4988-b6f9-c9babff54817-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:11.857061 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.857025 4918 generic.go:356] "Generic (PLEG): container finished" podID="942fc70c-abe0-4988-b6f9-c9babff54817" containerID="b040b847214a2009d427e3ea93b4bd146323adf4602fb0033ea9aab279e6e353" exitCode=0 Nov 03 20:36:11.857287 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.857081 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" Nov 03 20:36:11.857287 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.857110 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" event={"ID":"942fc70c-abe0-4988-b6f9-c9babff54817","Type":"ContainerDied","Data":"b040b847214a2009d427e3ea93b4bd146323adf4602fb0033ea9aab279e6e353"} Nov 03 20:36:11.857287 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.857170 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4" event={"ID":"942fc70c-abe0-4988-b6f9-c9babff54817","Type":"ContainerDied","Data":"b3dc8323fed4d52e2da8b073e54f036533821b3d5ed4bf8c69f1b7aa2017cc9e"} Nov 03 20:36:11.857287 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.857200 4918 scope.go:122] "RemoveContainer" containerID="b040b847214a2009d427e3ea93b4bd146323adf4602fb0033ea9aab279e6e353" Nov 03 20:36:11.866390 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.866371 4918 scope.go:122] "RemoveContainer" containerID="b040b847214a2009d427e3ea93b4bd146323adf4602fb0033ea9aab279e6e353" Nov 03 20:36:11.866645 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:36:11.866624 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b040b847214a2009d427e3ea93b4bd146323adf4602fb0033ea9aab279e6e353\": container with ID starting with b040b847214a2009d427e3ea93b4bd146323adf4602fb0033ea9aab279e6e353 not found: ID does not exist" containerID="b040b847214a2009d427e3ea93b4bd146323adf4602fb0033ea9aab279e6e353" Nov 03 20:36:11.866698 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.866656 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b040b847214a2009d427e3ea93b4bd146323adf4602fb0033ea9aab279e6e353"} err="failed to get container status \"b040b847214a2009d427e3ea93b4bd146323adf4602fb0033ea9aab279e6e353\": rpc error: code = NotFound desc = could not find container \"b040b847214a2009d427e3ea93b4bd146323adf4602fb0033ea9aab279e6e353\": container with ID starting with b040b847214a2009d427e3ea93b4bd146323adf4602fb0033ea9aab279e6e353 not found: ID does not exist" Nov 03 20:36:11.879124 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.879096 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4"] Nov 03 20:36:11.886854 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:11.886831 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-qddp4"] Nov 03 20:36:13.213389 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:13.213353 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="942fc70c-abe0-4988-b6f9-c9babff54817" path="/var/lib/kubelet/pods/942fc70c-abe0-4988-b6f9-c9babff54817/volumes" Nov 03 20:36:16.611033 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:16.610993 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x"] Nov 03 20:36:21.611836 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:21.611788 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x" podUID="0c47f933-d238-4576-b42e-36deda6d8528" containerName="payload-processing" containerID="cri-o://a5111b615af2f7e9279ca27c21a8ab4bbf48e5f340a84b3b921940033d9ac16b" gracePeriod=25 Nov 03 20:36:21.865358 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:21.865289 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x" Nov 03 20:36:21.918120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:21.918084 4918 generic.go:356] "Generic (PLEG): container finished" podID="0c47f933-d238-4576-b42e-36deda6d8528" containerID="a5111b615af2f7e9279ca27c21a8ab4bbf48e5f340a84b3b921940033d9ac16b" exitCode=0 Nov 03 20:36:21.918289 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:21.918134 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x" Nov 03 20:36:21.918289 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:21.918167 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x" event={"ID":"0c47f933-d238-4576-b42e-36deda6d8528","Type":"ContainerDied","Data":"a5111b615af2f7e9279ca27c21a8ab4bbf48e5f340a84b3b921940033d9ac16b"} Nov 03 20:36:21.918289 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:21.918236 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x" event={"ID":"0c47f933-d238-4576-b42e-36deda6d8528","Type":"ContainerDied","Data":"77efe7a70e433bd8c748de89bc6e8f40ca332c8c9165f1b26349171df6e30e38"} Nov 03 20:36:21.918289 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:21.918267 4918 scope.go:122] "RemoveContainer" containerID="a5111b615af2f7e9279ca27c21a8ab4bbf48e5f340a84b3b921940033d9ac16b" Nov 03 20:36:21.919388 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:21.919314 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c47f933-d238-4576-b42e-36deda6d8528-kube-api-access-52hwd\" (UniqueName: \"kubernetes.io/projected/0c47f933-d238-4576-b42e-36deda6d8528-kube-api-access-52hwd\") pod \"0c47f933-d238-4576-b42e-36deda6d8528\" (UID: \"0c47f933-d238-4576-b42e-36deda6d8528\") " Nov 03 20:36:21.919471 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:21.919391 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c47f933-d238-4576-b42e-36deda6d8528-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c47f933-d238-4576-b42e-36deda6d8528-plugins-config-volume\") pod \"0c47f933-d238-4576-b42e-36deda6d8528\" (UID: \"0c47f933-d238-4576-b42e-36deda6d8528\") " Nov 03 20:36:21.919783 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:21.919754 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c47f933-d238-4576-b42e-36deda6d8528-plugins-config-volume" pod "0c47f933-d238-4576-b42e-36deda6d8528" (UID: "0c47f933-d238-4576-b42e-36deda6d8528"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:21.922063 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:21.922026 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c47f933-d238-4576-b42e-36deda6d8528-kube-api-access-52hwd" pod "0c47f933-d238-4576-b42e-36deda6d8528" (UID: "0c47f933-d238-4576-b42e-36deda6d8528"). InnerVolumeSpecName "kube-api-access-52hwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:21.935767 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:21.935746 4918 scope.go:122] "RemoveContainer" containerID="a5111b615af2f7e9279ca27c21a8ab4bbf48e5f340a84b3b921940033d9ac16b" Nov 03 20:36:21.936034 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:36:21.936012 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5111b615af2f7e9279ca27c21a8ab4bbf48e5f340a84b3b921940033d9ac16b\": container with ID starting with a5111b615af2f7e9279ca27c21a8ab4bbf48e5f340a84b3b921940033d9ac16b not found: ID does not exist" containerID="a5111b615af2f7e9279ca27c21a8ab4bbf48e5f340a84b3b921940033d9ac16b" Nov 03 20:36:21.936082 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:21.936047 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5111b615af2f7e9279ca27c21a8ab4bbf48e5f340a84b3b921940033d9ac16b"} err="failed to get container status \"a5111b615af2f7e9279ca27c21a8ab4bbf48e5f340a84b3b921940033d9ac16b\": rpc error: code = NotFound desc = could not find container \"a5111b615af2f7e9279ca27c21a8ab4bbf48e5f340a84b3b921940033d9ac16b\": container with ID starting with a5111b615af2f7e9279ca27c21a8ab4bbf48e5f340a84b3b921940033d9ac16b not found: ID does not exist" Nov 03 20:36:22.021331 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:22.021297 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52hwd\" (UniqueName: \"kubernetes.io/projected/0c47f933-d238-4576-b42e-36deda6d8528-kube-api-access-52hwd\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:22.021331 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:22.021324 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c47f933-d238-4576-b42e-36deda6d8528-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:22.241204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:22.241160 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x"] Nov 03 20:36:22.245364 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:22.245341 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b1b190-7bc666c5c6-cxq8x"] Nov 03 20:36:23.213860 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:23.213821 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c47f933-d238-4576-b42e-36deda6d8528" path="/var/lib/kubelet/pods/0c47f933-d238-4576-b42e-36deda6d8528/volumes" Nov 03 20:36:28.209160 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:36:28.209120 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:36:39.134101 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:39.133999 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v"] Nov 03 20:36:39.134699 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:39.134290 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" podUID="eba28d4c-a907-4b6f-8d20-0a1baade4c58" containerName="istio-proxy" containerID="cri-o://d16e485a9994e28ac70118f08689a0b64a93b4584d763bac507797d5e86a77b5" gracePeriod=30 Nov 03 20:36:42.746014 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:42.745972 4918 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:36:42.746486 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:42.746037 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" podUID="eba28d4c-a907-4b6f-8d20-0a1baade4c58" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:36:44.672747 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.672392 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:36:44.812839 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.812804 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba28d4c-a907-4b6f-8d20-0a1baade4c58-kube-api-access-68ft7\" (UniqueName: \"kubernetes.io/projected/eba28d4c-a907-4b6f-8d20-0a1baade4c58-kube-api-access-68ft7\") pod \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " Nov 03 20:36:44.813040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.812865 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-envoy\") pod \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " Nov 03 20:36:44.813040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.812902 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-token\" (UniqueName: \"kubernetes.io/projected/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-token\") pod \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " Nov 03 20:36:44.813040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.812934 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istiod-ca-cert\") pod \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " Nov 03 20:36:44.813040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.812959 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-credential-socket\") pod \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " Nov 03 20:36:44.813040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.813016 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-podinfo\") pod \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " Nov 03 20:36:44.813040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.813040 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-workload-certs\") pod \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " Nov 03 20:36:44.813422 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.813071 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-workload-socket\") pod \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " Nov 03 20:36:44.813422 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.813125 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-data\") pod \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\" (UID: \"eba28d4c-a907-4b6f-8d20-0a1baade4c58\") " Nov 03 20:36:44.813538 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.813480 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istiod-ca-cert" pod "eba28d4c-a907-4b6f-8d20-0a1baade4c58" (UID: "eba28d4c-a907-4b6f-8d20-0a1baade4c58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:44.813538 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.813515 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-workload-certs" pod "eba28d4c-a907-4b6f-8d20-0a1baade4c58" (UID: "eba28d4c-a907-4b6f-8d20-0a1baade4c58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:44.813661 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.813539 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-data" pod "eba28d4c-a907-4b6f-8d20-0a1baade4c58" (UID: "eba28d4c-a907-4b6f-8d20-0a1baade4c58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:44.813738 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.813703 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-workload-socket" pod "eba28d4c-a907-4b6f-8d20-0a1baade4c58" (UID: "eba28d4c-a907-4b6f-8d20-0a1baade4c58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:44.813795 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.813715 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-credential-socket" pod "eba28d4c-a907-4b6f-8d20-0a1baade4c58" (UID: "eba28d4c-a907-4b6f-8d20-0a1baade4c58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:44.815330 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.815301 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-envoy" pod "eba28d4c-a907-4b6f-8d20-0a1baade4c58" (UID: "eba28d4c-a907-4b6f-8d20-0a1baade4c58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:44.815486 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.815449 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-podinfo" pod "eba28d4c-a907-4b6f-8d20-0a1baade4c58" (UID: "eba28d4c-a907-4b6f-8d20-0a1baade4c58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:36:44.815656 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.815641 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-token" pod "eba28d4c-a907-4b6f-8d20-0a1baade4c58" (UID: "eba28d4c-a907-4b6f-8d20-0a1baade4c58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:44.817504 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.817480 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba28d4c-a907-4b6f-8d20-0a1baade4c58-kube-api-access-68ft7" pod "eba28d4c-a907-4b6f-8d20-0a1baade4c58" (UID: "eba28d4c-a907-4b6f-8d20-0a1baade4c58"). InnerVolumeSpecName "kube-api-access-68ft7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:44.914767 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.914668 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.914767 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.914703 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68ft7\" (UniqueName: \"kubernetes.io/projected/eba28d4c-a907-4b6f-8d20-0a1baade4c58-kube-api-access-68ft7\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.914767 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.914714 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.914767 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.914723 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.914767 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.914731 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.914767 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.914740 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.914767 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.914749 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba28d4c-a907-4b6f-8d20-0a1baade4c58-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.914767 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.914758 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.914767 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:44.914775 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba28d4c-a907-4b6f-8d20-0a1baade4c58-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:45.080202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:45.079405 4918 generic.go:356] "Generic (PLEG): container finished" podID="eba28d4c-a907-4b6f-8d20-0a1baade4c58" containerID="d16e485a9994e28ac70118f08689a0b64a93b4584d763bac507797d5e86a77b5" exitCode=0 Nov 03 20:36:45.080202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:45.079586 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" event={"ID":"eba28d4c-a907-4b6f-8d20-0a1baade4c58","Type":"ContainerDied","Data":"d16e485a9994e28ac70118f08689a0b64a93b4584d763bac507797d5e86a77b5"} Nov 03 20:36:45.080202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:45.079680 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" event={"ID":"eba28d4c-a907-4b6f-8d20-0a1baade4c58","Type":"ContainerDied","Data":"0e47afcf75d86241e54ed9cc1aa4257a853d529832eacff985e69a31636b6bdc"} Nov 03 20:36:45.080202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:45.079713 4918 scope.go:122] "RemoveContainer" containerID="d16e485a9994e28ac70118f08689a0b64a93b4584d763bac507797d5e86a77b5" Nov 03 20:36:45.080202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:45.079883 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v" Nov 03 20:36:45.104255 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:45.104230 4918 scope.go:122] "RemoveContainer" containerID="d16e485a9994e28ac70118f08689a0b64a93b4584d763bac507797d5e86a77b5" Nov 03 20:36:45.104522 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:36:45.104500 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d16e485a9994e28ac70118f08689a0b64a93b4584d763bac507797d5e86a77b5\": container with ID starting with d16e485a9994e28ac70118f08689a0b64a93b4584d763bac507797d5e86a77b5 not found: ID does not exist" containerID="d16e485a9994e28ac70118f08689a0b64a93b4584d763bac507797d5e86a77b5" Nov 03 20:36:45.104565 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:45.104534 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d16e485a9994e28ac70118f08689a0b64a93b4584d763bac507797d5e86a77b5"} err="failed to get container status \"d16e485a9994e28ac70118f08689a0b64a93b4584d763bac507797d5e86a77b5\": rpc error: code = NotFound desc = could not find container \"d16e485a9994e28ac70118f08689a0b64a93b4584d763bac507797d5e86a77b5\": container with ID starting with d16e485a9994e28ac70118f08689a0b64a93b4584d763bac507797d5e86a77b5 not found: ID does not exist" Nov 03 20:36:45.115757 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:45.115731 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v"] Nov 03 20:36:45.123086 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:45.123049 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-f6c4v"] Nov 03 20:36:45.213661 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:45.213578 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eba28d4c-a907-4b6f-8d20-0a1baade4c58" path="/var/lib/kubelet/pods/eba28d4c-a907-4b6f-8d20-0a1baade4c58/volumes" Nov 03 20:36:57.561399 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:57.561363 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:36:57.568869 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:36:57.568841 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:36:58.207744 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:36:58.207707 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:37:00.043762 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:00.043730 4918 scope.go:122] "RemoveContainer" containerID="09e6b0aabba3904104d1daa6d4f6a4abf4997d7255b3b3f190b9860a58c0a5c2" Nov 03 20:37:13.114270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.114229 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7"] Nov 03 20:37:13.114947 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.114916 4918 cpu_manager.go:414] "containerMap: removing container" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" Nov 03 20:37:13.114947 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.114946 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" Nov 03 20:37:13.115096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.114962 4918 cpu_manager.go:414] "containerMap: removing container" podUID="81a01d22-8738-4955-9dc5-73d769266ddc" containerName="cleanup" Nov 03 20:37:13.115096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.114971 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a01d22-8738-4955-9dc5-73d769266ddc" containerName="cleanup" Nov 03 20:37:13.115096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.114986 4918 cpu_manager.go:414] "containerMap: removing container" podUID="645fc172-d7a8-48a6-93ec-6c9ed0fc0304" containerName="istio-proxy" Nov 03 20:37:13.115096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.114994 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645fc172-d7a8-48a6-93ec-6c9ed0fc0304" containerName="istio-proxy" Nov 03 20:37:13.115096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115005 4918 cpu_manager.go:414] "containerMap: removing container" podUID="3e465e5f-cb99-475a-848a-a182ceccc56e" containerName="istio-proxy" Nov 03 20:37:13.115096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115014 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e465e5f-cb99-475a-848a-a182ceccc56e" containerName="istio-proxy" Nov 03 20:37:13.115096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115033 4918 cpu_manager.go:414] "containerMap: removing container" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" Nov 03 20:37:13.115096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115041 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" Nov 03 20:37:13.115096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115055 4918 cpu_manager.go:414] "containerMap: removing container" podUID="7a2653fa-4e55-48c8-a756-1f24c4c8412a" containerName="cleanup" Nov 03 20:37:13.115096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115063 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2653fa-4e55-48c8-a756-1f24c4c8412a" containerName="cleanup" Nov 03 20:37:13.115096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115079 4918 cpu_manager.go:414] "containerMap: removing container" podUID="b92265eb-94c4-42c6-b2c0-b22feab2eb07" containerName="istio-proxy" Nov 03 20:37:13.115096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115087 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92265eb-94c4-42c6-b2c0-b22feab2eb07" containerName="istio-proxy" Nov 03 20:37:13.115096 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115097 4918 cpu_manager.go:414] "containerMap: removing container" podUID="0c47f933-d238-4576-b42e-36deda6d8528" containerName="payload-processing" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115105 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c47f933-d238-4576-b42e-36deda6d8528" containerName="payload-processing" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115116 4918 cpu_manager.go:414] "containerMap: removing container" podUID="c4cccb65-6424-4e0b-85bf-6f1fba8d5572" containerName="cleanup" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115123 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4cccb65-6424-4e0b-85bf-6f1fba8d5572" containerName="cleanup" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115140 4918 cpu_manager.go:414] "containerMap: removing container" podUID="e2ae2662-2a94-411f-a9e1-a0fb1452f3ae" containerName="payload-pre-processing" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115148 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2ae2662-2a94-411f-a9e1-a0fb1452f3ae" containerName="payload-pre-processing" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115159 4918 cpu_manager.go:414] "containerMap: removing container" podUID="af357454-5d01-46c9-a1ef-bf09169443ee" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115168 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af357454-5d01-46c9-a1ef-bf09169443ee" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115196 4918 cpu_manager.go:414] "containerMap: removing container" podUID="5120ad5e-81f7-43c1-9e1d-f0407339bb0c" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115203 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5120ad5e-81f7-43c1-9e1d-f0407339bb0c" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115215 4918 cpu_manager.go:414] "containerMap: removing container" podUID="22993cbd-6df5-493a-b19a-3f1269dbc57c" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115222 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22993cbd-6df5-493a-b19a-3f1269dbc57c" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115233 4918 cpu_manager.go:414] "containerMap: removing container" podUID="eba28d4c-a907-4b6f-8d20-0a1baade4c58" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115240 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eba28d4c-a907-4b6f-8d20-0a1baade4c58" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115253 4918 cpu_manager.go:414] "containerMap: removing container" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115262 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115274 4918 cpu_manager.go:414] "containerMap: removing container" podUID="d7646487-52b3-4fd9-8781-2d440af503a2" containerName="maas-api" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115280 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7646487-52b3-4fd9-8781-2d440af503a2" containerName="maas-api" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115297 4918 cpu_manager.go:414] "containerMap: removing container" podUID="942fc70c-abe0-4988-b6f9-c9babff54817" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115304 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942fc70c-abe0-4988-b6f9-c9babff54817" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115314 4918 cpu_manager.go:414] "containerMap: removing container" podUID="3e465e5f-cb99-475a-848a-a182ceccc56e" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115321 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e465e5f-cb99-475a-848a-a182ceccc56e" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115330 4918 cpu_manager.go:414] "containerMap: removing container" podUID="be8a9e79-44b3-4610-acbc-ee4dd69c121f" containerName="payload-processing" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115338 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be8a9e79-44b3-4610-acbc-ee4dd69c121f" containerName="payload-processing" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115346 4918 cpu_manager.go:414] "containerMap: removing container" podUID="38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c" containerName="payload-processing" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115351 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c" containerName="payload-processing" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115361 4918 cpu_manager.go:414] "containerMap: removing container" podUID="935c467f-7d4e-4ea6-9403-08b0079f774c" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115365 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="935c467f-7d4e-4ea6-9403-08b0079f774c" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115372 4918 cpu_manager.go:414] "containerMap: removing container" podUID="0d724b00-97ad-4d6f-85a6-3a273b5042f1" containerName="payload-pre-processing" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115376 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d724b00-97ad-4d6f-85a6-3a273b5042f1" containerName="payload-pre-processing" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115386 4918 cpu_manager.go:414] "containerMap: removing container" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115390 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115408 4918 cpu_manager.go:414] "containerMap: removing container" podUID="4aaf5a9a-91e9-48d8-b69d-97de7d99d51a" containerName="main" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115414 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aaf5a9a-91e9-48d8-b69d-97de7d99d51a" containerName="main" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115421 4918 cpu_manager.go:414] "containerMap: removing container" podUID="af357454-5d01-46c9-a1ef-bf09169443ee" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115425 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af357454-5d01-46c9-a1ef-bf09169443ee" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115433 4918 cpu_manager.go:414] "containerMap: removing container" podUID="a1c85e9e-0e22-4860-8859-b36713dcbcfc" containerName="maas-api" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115437 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c85e9e-0e22-4860-8859-b36713dcbcfc" containerName="maas-api" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115443 4918 cpu_manager.go:414] "containerMap: removing container" podUID="54a415e5-f853-4579-9f3f-0c3c7a9216e5" containerName="main" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115448 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a415e5-f853-4579-9f3f-0c3c7a9216e5" containerName="main" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115456 4918 cpu_manager.go:414] "containerMap: removing container" podUID="2228ed1d-945a-4b33-ba98-ee8b52cbf902" containerName="cleanup" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115460 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2228ed1d-945a-4b33-ba98-ee8b52cbf902" containerName="cleanup" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115471 4918 cpu_manager.go:414] "containerMap: removing container" podUID="077732c1-4eda-4ce4-8afe-44e00e632df9" containerName="maas-api" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115475 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="077732c1-4eda-4ce4-8afe-44e00e632df9" containerName="maas-api" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115481 4918 cpu_manager.go:414] "containerMap: removing container" podUID="4e08c21d-00c5-4769-87fc-253cb19a7c85" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115485 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e08c21d-00c5-4769-87fc-253cb19a7c85" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115492 4918 cpu_manager.go:414] "containerMap: removing container" podUID="1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115496 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115588 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c85e9e-0e22-4860-8859-b36713dcbcfc" containerName="maas-api" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115603 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a415e5-f853-4579-9f3f-0c3c7a9216e5" containerName="main" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115609 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2228ed1d-945a-4b33-ba98-ee8b52cbf902" containerName="cleanup" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115615 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="077732c1-4eda-4ce4-8afe-44e00e632df9" containerName="maas-api" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115623 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e08c21d-00c5-4769-87fc-253cb19a7c85" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115629 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4cccb65-6424-4e0b-85bf-6f1fba8d5572" containerName="cleanup" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115638 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="189c26eb-8160-4721-9775-f27379339f50" containerName="main" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115646 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a01d22-8738-4955-9dc5-73d769266ddc" containerName="cleanup" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115655 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="645fc172-d7a8-48a6-93ec-6c9ed0fc0304" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115663 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="645fc172-d7a8-48a6-93ec-6c9ed0fc0304" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115669 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e465e5f-cb99-475a-848a-a182ceccc56e" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115675 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="812fc025-3310-47d1-80dd-72b73ef4494e" containerName="main" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115681 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92265eb-94c4-42c6-b2c0-b22feab2eb07" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115687 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c47f933-d238-4576-b42e-36deda6d8528" containerName="payload-processing" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115693 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2ae2662-2a94-411f-a9e1-a0fb1452f3ae" containerName="payload-pre-processing" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115698 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="af357454-5d01-46c9-a1ef-bf09169443ee" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115706 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc721d0a-faab-45c5-8476-9aec206a65c3" containerName="cleanup" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115715 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5120ad5e-81f7-43c1-9e1d-f0407339bb0c" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115724 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="22993cbd-6df5-493a-b19a-3f1269dbc57c" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115732 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="eba28d4c-a907-4b6f-8d20-0a1baade4c58" containerName="istio-proxy" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115740 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="bad512a6-de2f-428d-a37a-e56eb56e15f6" containerName="main" Nov 03 20:37:13.115714 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115750 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d98bfc5-35c3-4925-a3d1-696dd0f90dd7" containerName="istio-proxy" Nov 03 20:37:13.118696 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115761 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7646487-52b3-4fd9-8781-2d440af503a2" containerName="maas-api" Nov 03 20:37:13.118696 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115769 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92265eb-94c4-42c6-b2c0-b22feab2eb07" containerName="istio-proxy" Nov 03 20:37:13.118696 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115778 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="942fc70c-abe0-4988-b6f9-c9babff54817" containerName="istio-proxy" Nov 03 20:37:13.118696 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115785 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb1ff44-4b90-47da-a252-d5771f160330" containerName="istio-proxy" Nov 03 20:37:13.118696 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115794 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e465e5f-cb99-475a-848a-a182ceccc56e" containerName="istio-proxy" Nov 03 20:37:13.118696 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115802 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="be8a9e79-44b3-4610-acbc-ee4dd69c121f" containerName="payload-processing" Nov 03 20:37:13.118696 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115810 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b48de8-eb8c-46cc-bfa4-6ef2a2bb0a4c" containerName="payload-processing" Nov 03 20:37:13.118696 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115819 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8f2527-d745-4a36-b656-594d782b0db3" containerName="istio-proxy" Nov 03 20:37:13.118696 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115827 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0832a6d3-76cd-4eb6-9879-c880f98a786f" containerName="cleanup" Nov 03 20:37:13.118696 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115836 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="935c467f-7d4e-4ea6-9403-08b0079f774c" containerName="istio-proxy" Nov 03 20:37:13.118696 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115845 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d724b00-97ad-4d6f-85a6-3a273b5042f1" containerName="payload-pre-processing" Nov 03 20:37:13.118696 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115855 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aaf5a9a-91e9-48d8-b69d-97de7d99d51a" containerName="main" Nov 03 20:37:13.118696 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115864 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe6c2991-de3a-4374-a5c3-63816a171924" containerName="main" Nov 03 20:37:13.118696 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.115873 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="af357454-5d01-46c9-a1ef-bf09169443ee" containerName="istio-proxy" Nov 03 20:37:13.120848 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.120828 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.131454 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.131425 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7"] Nov 03 20:37:13.206318 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.206284 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.206505 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.206325 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.206505 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.206346 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.206505 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.206435 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsbvw\" (UniqueName: \"kubernetes.io/projected/67ba957b-0d42-473f-b2fb-c6c5fbea9885-kube-api-access-xsbvw\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.206679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.206506 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.206679 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.206636 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.206804 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.206687 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.206804 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.206792 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.206909 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.206836 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.307744 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.307705 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.307926 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.307757 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.307926 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.307874 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.308048 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.307948 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.308048 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.307997 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.308168 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.308051 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.308168 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.308086 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.308168 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.308114 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.308168 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.308149 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsbvw\" (UniqueName: \"kubernetes.io/projected/67ba957b-0d42-473f-b2fb-c6c5fbea9885-kube-api-access-xsbvw\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.308484 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.308168 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.308484 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.308293 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.308602 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.308504 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.308663 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.308614 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.308770 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.308745 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.310333 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.310307 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.310438 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.310309 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.320843 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.320819 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.321030 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.321010 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsbvw\" (UniqueName: \"kubernetes.io/projected/67ba957b-0d42-473f-b2fb-c6c5fbea9885-kube-api-access-xsbvw\") pod \"maas-default-gateway-openshift-default-77f556d647-xwdt7\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.437305 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.437212 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:13.741360 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.741326 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7"] Nov 03 20:37:13.743413 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:37:13.743384 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ba957b_0d42_473f_b2fb_c6c5fbea9885.slice/crio-e72dad55967f6fc55526715df191f34871d5a9804bf958b3ca0c7ec779fa7766 WatchSource:0}: Error finding container e72dad55967f6fc55526715df191f34871d5a9804bf958b3ca0c7ec779fa7766: Status 404 returned error can't find the container with id e72dad55967f6fc55526715df191f34871d5a9804bf958b3ca0c7ec779fa7766 Nov 03 20:37:13.746067 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.746031 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:37:13.746157 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.746114 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:37:13.746157 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:13.746145 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:37:14.248743 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:14.248701 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" event={"ID":"67ba957b-0d42-473f-b2fb-c6c5fbea9885","Type":"ContainerStarted","Data":"2205fca82fe4e010a2e336d348e30c7a5fe774ed231504e4a292c2945f829ae9"} Nov 03 20:37:14.249256 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:14.248765 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" event={"ID":"67ba957b-0d42-473f-b2fb-c6c5fbea9885","Type":"ContainerStarted","Data":"e72dad55967f6fc55526715df191f34871d5a9804bf958b3ca0c7ec779fa7766"} Nov 03 20:37:14.275356 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:14.275304 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" podStartSLOduration=1.275289799 podStartE2EDuration="1.275289799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:14.273126925 +0000 UTC m=+2417.204994438" watchObservedRunningTime="2026-11-03 20:37:14.275289799 +0000 UTC m=+2417.207157310" Nov 03 20:37:14.445214 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:14.444547 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:14.445214 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:14.444709 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xwdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 20:37:14.445214 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:14.444765 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" podUID="67ba957b-0d42-473f-b2fb-c6c5fbea9885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 20:37:15.441624 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:15.441580 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xwdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 20:37:15.442286 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:15.442237 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" podUID="67ba957b-0d42-473f-b2fb-c6c5fbea9885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 20:37:16.439201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:16.437491 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xwdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 20:37:16.439201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:16.437554 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" podUID="67ba957b-0d42-473f-b2fb-c6c5fbea9885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 20:37:17.443623 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:17.443584 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xwdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 20:37:17.444048 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:17.443641 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" podUID="67ba957b-0d42-473f-b2fb-c6c5fbea9885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 20:37:18.438202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:18.437574 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xwdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 20:37:18.438202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:18.437636 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" podUID="67ba957b-0d42-473f-b2fb-c6c5fbea9885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 20:37:19.443643 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:19.443600 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xwdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 20:37:19.444020 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:19.443662 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" podUID="67ba957b-0d42-473f-b2fb-c6c5fbea9885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 20:37:20.438565 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:20.438519 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xwdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 20:37:20.438792 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:20.438581 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" podUID="67ba957b-0d42-473f-b2fb-c6c5fbea9885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 20:37:21.932937 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:21.932888 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:21.933413 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:21.933226 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:21.934282 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:37:21.934257 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:37:28.207997 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:37:28.207964 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:37:58.208277 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:37:58.208242 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:38:22.979705 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:22.979625 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj"] Nov 03 20:38:22.980276 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:22.979838 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" podUID="673a0598-869d-446b-af9c-6a2e7b0636aa" containerName="discovery" containerID="cri-o://72aa70dbdc42d685768cc6b48347a0f1306dc2227cd3eca79f4a7f0f5718a1a3" gracePeriod=30 Nov 03 20:38:23.254268 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.254238 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:38:23.374254 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.374219 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-token\" (UniqueName: \"kubernetes.io/projected/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-token\") pod \"673a0598-869d-446b-af9c-6a2e7b0636aa\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " Nov 03 20:38:23.374254 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.374261 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/673a0598-869d-446b-af9c-6a2e7b0636aa-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/673a0598-869d-446b-af9c-6a2e7b0636aa-local-certs\") pod \"673a0598-869d-446b-af9c-6a2e7b0636aa\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " Nov 03 20:38:23.374518 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.374314 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-kubeconfig\") pod \"673a0598-869d-446b-af9c-6a2e7b0636aa\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " Nov 03 20:38:23.374518 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.374346 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-csr-ca-configmap\") pod \"673a0598-869d-446b-af9c-6a2e7b0636aa\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " Nov 03 20:38:23.374518 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.374373 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-cacerts\" (UniqueName: \"kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-cacerts\") pod \"673a0598-869d-446b-af9c-6a2e7b0636aa\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " Nov 03 20:38:23.374518 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.374403 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/673a0598-869d-446b-af9c-6a2e7b0636aa-kube-api-access-ll64r\" (UniqueName: \"kubernetes.io/projected/673a0598-869d-446b-af9c-6a2e7b0636aa-kube-api-access-ll64r\") pod \"673a0598-869d-446b-af9c-6a2e7b0636aa\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " Nov 03 20:38:23.374518 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.374438 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-csr-dns-cert\") pod \"673a0598-869d-446b-af9c-6a2e7b0636aa\" (UID: \"673a0598-869d-446b-af9c-6a2e7b0636aa\") " Nov 03 20:38:23.374822 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.374783 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-csr-ca-configmap" pod "673a0598-869d-446b-af9c-6a2e7b0636aa" (UID: "673a0598-869d-446b-af9c-6a2e7b0636aa"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:23.376864 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.376679 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-kubeconfig" pod "673a0598-869d-446b-af9c-6a2e7b0636aa" (UID: "673a0598-869d-446b-af9c-6a2e7b0636aa"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:23.377004 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.376922 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-csr-dns-cert" pod "673a0598-869d-446b-af9c-6a2e7b0636aa" (UID: "673a0598-869d-446b-af9c-6a2e7b0636aa"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:23.377059 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.377031 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-cacerts" pod "673a0598-869d-446b-af9c-6a2e7b0636aa" (UID: "673a0598-869d-446b-af9c-6a2e7b0636aa"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:23.377150 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.377121 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-token" pod "673a0598-869d-446b-af9c-6a2e7b0636aa" (UID: "673a0598-869d-446b-af9c-6a2e7b0636aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:23.377246 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.377129 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673a0598-869d-446b-af9c-6a2e7b0636aa-local-certs" pod "673a0598-869d-446b-af9c-6a2e7b0636aa" (UID: "673a0598-869d-446b-af9c-6a2e7b0636aa"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:23.377293 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.377246 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673a0598-869d-446b-af9c-6a2e7b0636aa-kube-api-access-ll64r" pod "673a0598-869d-446b-af9c-6a2e7b0636aa" (UID: "673a0598-869d-446b-af9c-6a2e7b0636aa"). InnerVolumeSpecName "kube-api-access-ll64r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:23.475353 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.475312 4918 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-kubeconfig\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:23.475353 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.475347 4918 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-csr-ca-configmap\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:23.475575 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.475362 4918 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-cacerts\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:23.475575 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.475376 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll64r\" (UniqueName: \"kubernetes.io/projected/673a0598-869d-446b-af9c-6a2e7b0636aa-kube-api-access-ll64r\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:23.475575 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.475388 4918 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-csr-dns-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:23.475575 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.475400 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/673a0598-869d-446b-af9c-6a2e7b0636aa-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:23.475575 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.475411 4918 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/673a0598-869d-446b-af9c-6a2e7b0636aa-local-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:23.717801 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.717760 4918 generic.go:356] "Generic (PLEG): container finished" podID="673a0598-869d-446b-af9c-6a2e7b0636aa" containerID="72aa70dbdc42d685768cc6b48347a0f1306dc2227cd3eca79f4a7f0f5718a1a3" exitCode=0 Nov 03 20:38:23.717974 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.717821 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" Nov 03 20:38:23.717974 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.717822 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" event={"ID":"673a0598-869d-446b-af9c-6a2e7b0636aa","Type":"ContainerDied","Data":"72aa70dbdc42d685768cc6b48347a0f1306dc2227cd3eca79f4a7f0f5718a1a3"} Nov 03 20:38:23.717974 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.717949 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj" event={"ID":"673a0598-869d-446b-af9c-6a2e7b0636aa","Type":"ContainerDied","Data":"447c5715205508fca086e04b94c4682dab649360214a16b415942e3d805341f4"} Nov 03 20:38:23.718108 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.717976 4918 scope.go:122] "RemoveContainer" containerID="72aa70dbdc42d685768cc6b48347a0f1306dc2227cd3eca79f4a7f0f5718a1a3" Nov 03 20:38:23.731448 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.731424 4918 scope.go:122] "RemoveContainer" containerID="72aa70dbdc42d685768cc6b48347a0f1306dc2227cd3eca79f4a7f0f5718a1a3" Nov 03 20:38:23.731764 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:38:23.731738 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72aa70dbdc42d685768cc6b48347a0f1306dc2227cd3eca79f4a7f0f5718a1a3\": container with ID starting with 72aa70dbdc42d685768cc6b48347a0f1306dc2227cd3eca79f4a7f0f5718a1a3 not found: ID does not exist" containerID="72aa70dbdc42d685768cc6b48347a0f1306dc2227cd3eca79f4a7f0f5718a1a3" Nov 03 20:38:23.731856 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.731777 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72aa70dbdc42d685768cc6b48347a0f1306dc2227cd3eca79f4a7f0f5718a1a3"} err="failed to get container status \"72aa70dbdc42d685768cc6b48347a0f1306dc2227cd3eca79f4a7f0f5718a1a3\": rpc error: code = NotFound desc = could not find container \"72aa70dbdc42d685768cc6b48347a0f1306dc2227cd3eca79f4a7f0f5718a1a3\": container with ID starting with 72aa70dbdc42d685768cc6b48347a0f1306dc2227cd3eca79f4a7f0f5718a1a3 not found: ID does not exist" Nov 03 20:38:23.742152 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.742125 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj"] Nov 03 20:38:23.746854 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:23.746826 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s4brj"] Nov 03 20:38:25.213319 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:25.213287 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="673a0598-869d-446b-af9c-6a2e7b0636aa" path="/var/lib/kubelet/pods/673a0598-869d-446b-af9c-6a2e7b0636aa/volumes" Nov 03 20:38:28.207699 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:38:28.207658 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:38:34.935348 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:34.935303 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8bvn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:38:34.936780 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:38:34.936739 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" podUID="ea819fe0-461b-4952-a40e-7a47e8f51580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:38:58.207751 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:38:58.207714 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:39:19.927371 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:39:19.927329 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8bvn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:39:19.928097 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:39:19.928062 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" podUID="ea819fe0-461b-4952-a40e-7a47e8f51580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:39:28.212056 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:39:28.212014 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:39:58.206929 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:39:58.206893 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:40:04.931819 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:40:04.931773 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8bvn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:40:04.932704 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:40:04.932664 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" podUID="ea819fe0-461b-4952-a40e-7a47e8f51580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:40:28.208486 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:40:28.208441 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:40:58.207820 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:40:58.207773 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:41:28.207376 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:41:28.207337 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:41:37.521259 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:41:37.521211 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-t7zgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:41:37.522641 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:41:37.522606 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-t7zgq" podUID="ea43d6b7-986b-4194-b9b3-b86e22e86544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:41:57.592924 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:41:57.592894 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:41:57.600904 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:41:57.600871 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:41:58.208699 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:41:58.208658 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:42:28.207351 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:42:28.207305 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:42:58.207251 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:42:58.207208 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:43:28.208291 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:43:28.208253 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:43:58.207035 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:43:58.206997 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:44:28.206951 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:44:28.206911 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:44:58.207808 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:44:58.207767 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:45:28.207888 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:45:28.207853 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:45:58.207794 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:45:58.207696 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:46:28.208017 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:46:28.207979 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:46:57.621159 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:46:57.621127 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:46:57.631215 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:46:57.631189 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:46:58.208084 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:46:58.208041 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:47:23.245085 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:23.245028 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zjhwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:47:23.245904 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:23.245867 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" podUID="da26a005-8c1e-46a4-8a17-93119a8a735d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:28.208238 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:47:28.208199 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:47:28.796318 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:28.796281 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7"] Nov 03 20:47:28.796704 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:28.796528 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" podUID="67ba957b-0d42-473f-b2fb-c6c5fbea9885" containerName="istio-proxy" containerID="cri-o://2205fca82fe4e010a2e336d348e30c7a5fe774ed231504e4a292c2945f829ae9" gracePeriod=30 Nov 03 20:47:28.798906 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:28.798880 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr"] Nov 03 20:47:28.799068 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:28.799047 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" podUID="da26a005-8c1e-46a4-8a17-93119a8a735d" containerName="istio-proxy" containerID="cri-o://ef21e567cda94fc66e98c4cfc1fc93a265404e8091928a8687dab1c20799410e" gracePeriod=30 Nov 03 20:47:34.064596 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.064567 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:47:34.090387 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.090360 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:47:34.175339 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.175298 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-envoy\") pod \"da26a005-8c1e-46a4-8a17-93119a8a735d\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " Nov 03 20:47:34.175540 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.175373 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ba957b-0d42-473f-b2fb-c6c5fbea9885-kube-api-access-xsbvw\" (UniqueName: \"kubernetes.io/projected/67ba957b-0d42-473f-b2fb-c6c5fbea9885-kube-api-access-xsbvw\") pod \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " Nov 03 20:47:34.175540 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.175401 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-workload-certs\") pod \"da26a005-8c1e-46a4-8a17-93119a8a735d\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " Nov 03 20:47:34.175540 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.175431 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-podinfo\") pod \"da26a005-8c1e-46a4-8a17-93119a8a735d\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " Nov 03 20:47:34.175540 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.175469 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-workload-certs\") pod \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " Nov 03 20:47:34.175540 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.175515 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-workload-socket\") pod \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " Nov 03 20:47:34.175822 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.175552 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-credential-socket\") pod \"da26a005-8c1e-46a4-8a17-93119a8a735d\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " Nov 03 20:47:34.175822 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.175601 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-token\" (UniqueName: \"kubernetes.io/projected/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-token\") pod \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " Nov 03 20:47:34.175822 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.175639 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-envoy\") pod \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " Nov 03 20:47:34.175822 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.175669 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-token\" (UniqueName: \"kubernetes.io/projected/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-token\") pod \"da26a005-8c1e-46a4-8a17-93119a8a735d\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " Nov 03 20:47:34.175822 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.175714 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-data\") pod \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " Nov 03 20:47:34.175822 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.175720 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-workload-certs" pod "da26a005-8c1e-46a4-8a17-93119a8a735d" (UID: "da26a005-8c1e-46a4-8a17-93119a8a735d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:34.175822 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.175747 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da26a005-8c1e-46a4-8a17-93119a8a735d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da26a005-8c1e-46a4-8a17-93119a8a735d-istiod-ca-cert\") pod \"da26a005-8c1e-46a4-8a17-93119a8a735d\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " Nov 03 20:47:34.176229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.175815 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-workload-certs" pod "67ba957b-0d42-473f-b2fb-c6c5fbea9885" (UID: "67ba957b-0d42-473f-b2fb-c6c5fbea9885"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:34.176229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.175835 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istiod-ca-cert\") pod \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " Nov 03 20:47:34.176229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.175884 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-workload-socket\") pod \"da26a005-8c1e-46a4-8a17-93119a8a735d\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " Nov 03 20:47:34.176229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.175915 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-credential-socket\") pod \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " Nov 03 20:47:34.176229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.175956 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-data\") pod \"da26a005-8c1e-46a4-8a17-93119a8a735d\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " Nov 03 20:47:34.176229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.176016 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da26a005-8c1e-46a4-8a17-93119a8a735d-kube-api-access-2fnsq\" (UniqueName: \"kubernetes.io/projected/da26a005-8c1e-46a4-8a17-93119a8a735d-kube-api-access-2fnsq\") pod \"da26a005-8c1e-46a4-8a17-93119a8a735d\" (UID: \"da26a005-8c1e-46a4-8a17-93119a8a735d\") " Nov 03 20:47:34.176229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.176068 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-podinfo\") pod \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\" (UID: \"67ba957b-0d42-473f-b2fb-c6c5fbea9885\") " Nov 03 20:47:34.176229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.176107 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-workload-socket" pod "67ba957b-0d42-473f-b2fb-c6c5fbea9885" (UID: "67ba957b-0d42-473f-b2fb-c6c5fbea9885"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:34.176229 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.176139 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da26a005-8c1e-46a4-8a17-93119a8a735d-istiod-ca-cert" pod "da26a005-8c1e-46a4-8a17-93119a8a735d" (UID: "da26a005-8c1e-46a4-8a17-93119a8a735d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:34.176771 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.176339 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-data" pod "67ba957b-0d42-473f-b2fb-c6c5fbea9885" (UID: "67ba957b-0d42-473f-b2fb-c6c5fbea9885"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:34.176771 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.176345 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-credential-socket" pod "da26a005-8c1e-46a4-8a17-93119a8a735d" (UID: "da26a005-8c1e-46a4-8a17-93119a8a735d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:34.176771 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.176514 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:34.176771 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.176534 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:34.176771 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.176549 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da26a005-8c1e-46a4-8a17-93119a8a735d-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:34.176771 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.176564 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:34.176771 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.176579 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:34.176771 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.176596 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:34.177222 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.176815 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-credential-socket" pod "67ba957b-0d42-473f-b2fb-c6c5fbea9885" (UID: "67ba957b-0d42-473f-b2fb-c6c5fbea9885"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:34.177222 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.177094 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istiod-ca-cert" pod "67ba957b-0d42-473f-b2fb-c6c5fbea9885" (UID: "67ba957b-0d42-473f-b2fb-c6c5fbea9885"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:34.177370 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.177351 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-workload-socket" pod "da26a005-8c1e-46a4-8a17-93119a8a735d" (UID: "da26a005-8c1e-46a4-8a17-93119a8a735d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:34.177533 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.177514 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-data" pod "da26a005-8c1e-46a4-8a17-93119a8a735d" (UID: "da26a005-8c1e-46a4-8a17-93119a8a735d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:34.178639 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.178577 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-envoy" pod "da26a005-8c1e-46a4-8a17-93119a8a735d" (UID: "da26a005-8c1e-46a4-8a17-93119a8a735d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:34.178639 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.178610 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-token" pod "da26a005-8c1e-46a4-8a17-93119a8a735d" (UID: "da26a005-8c1e-46a4-8a17-93119a8a735d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:34.178786 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.178642 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-podinfo" pod "da26a005-8c1e-46a4-8a17-93119a8a735d" (UID: "da26a005-8c1e-46a4-8a17-93119a8a735d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:47:34.178786 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.178646 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-podinfo" pod "67ba957b-0d42-473f-b2fb-c6c5fbea9885" (UID: "67ba957b-0d42-473f-b2fb-c6c5fbea9885"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:47:34.178945 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.178924 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ba957b-0d42-473f-b2fb-c6c5fbea9885-kube-api-access-xsbvw" pod "67ba957b-0d42-473f-b2fb-c6c5fbea9885" (UID: "67ba957b-0d42-473f-b2fb-c6c5fbea9885"). InnerVolumeSpecName "kube-api-access-xsbvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:34.179116 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.179093 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-token" pod "67ba957b-0d42-473f-b2fb-c6c5fbea9885" (UID: "67ba957b-0d42-473f-b2fb-c6c5fbea9885"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:34.179300 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.179276 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-envoy" pod "67ba957b-0d42-473f-b2fb-c6c5fbea9885" (UID: "67ba957b-0d42-473f-b2fb-c6c5fbea9885"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:34.179488 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.179471 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da26a005-8c1e-46a4-8a17-93119a8a735d-kube-api-access-2fnsq" pod "da26a005-8c1e-46a4-8a17-93119a8a735d" (UID: "da26a005-8c1e-46a4-8a17-93119a8a735d"). InnerVolumeSpecName "kube-api-access-2fnsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:34.277804 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.277764 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:34.277804 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.277797 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:34.277804 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.277810 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:34.278057 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.277823 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:34.278057 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.277837 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ba957b-0d42-473f-b2fb-c6c5fbea9885-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:34.278057 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.277848 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:34.278057 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.277860 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fnsq\" (UniqueName: \"kubernetes.io/projected/da26a005-8c1e-46a4-8a17-93119a8a735d-kube-api-access-2fnsq\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:34.278057 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.277874 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:34.278057 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.277886 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:34.278057 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.277898 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsbvw\" (UniqueName: \"kubernetes.io/projected/67ba957b-0d42-473f-b2fb-c6c5fbea9885-kube-api-access-xsbvw\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:34.278057 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.277910 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da26a005-8c1e-46a4-8a17-93119a8a735d-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:34.278057 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.277922 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ba957b-0d42-473f-b2fb-c6c5fbea9885-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:34.630167 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.630132 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5"] Nov 03 20:47:34.630596 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.630576 4918 cpu_manager.go:414] "containerMap: removing container" podUID="645fc172-d7a8-48a6-93ec-6c9ed0fc0304" containerName="istio-proxy" Nov 03 20:47:34.630596 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.630593 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645fc172-d7a8-48a6-93ec-6c9ed0fc0304" containerName="istio-proxy" Nov 03 20:47:34.630746 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.630606 4918 cpu_manager.go:414] "containerMap: removing container" podUID="67ba957b-0d42-473f-b2fb-c6c5fbea9885" containerName="istio-proxy" Nov 03 20:47:34.630746 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.630612 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67ba957b-0d42-473f-b2fb-c6c5fbea9885" containerName="istio-proxy" Nov 03 20:47:34.630746 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.630627 4918 cpu_manager.go:414] "containerMap: removing container" podUID="b92265eb-94c4-42c6-b2c0-b22feab2eb07" containerName="istio-proxy" Nov 03 20:47:34.630746 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.630635 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92265eb-94c4-42c6-b2c0-b22feab2eb07" containerName="istio-proxy" Nov 03 20:47:34.630746 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.630646 4918 cpu_manager.go:414] "containerMap: removing container" podUID="da26a005-8c1e-46a4-8a17-93119a8a735d" containerName="istio-proxy" Nov 03 20:47:34.630746 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.630651 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da26a005-8c1e-46a4-8a17-93119a8a735d" containerName="istio-proxy" Nov 03 20:47:34.630746 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.630672 4918 cpu_manager.go:414] "containerMap: removing container" podUID="673a0598-869d-446b-af9c-6a2e7b0636aa" containerName="discovery" Nov 03 20:47:34.630746 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.630677 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="673a0598-869d-446b-af9c-6a2e7b0636aa" containerName="discovery" Nov 03 20:47:34.630746 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.630746 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="da26a005-8c1e-46a4-8a17-93119a8a735d" containerName="istio-proxy" Nov 03 20:47:34.631031 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.630758 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="673a0598-869d-446b-af9c-6a2e7b0636aa" containerName="discovery" Nov 03 20:47:34.631031 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.630768 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="67ba957b-0d42-473f-b2fb-c6c5fbea9885" containerName="istio-proxy" Nov 03 20:47:34.635120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.635102 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5" Nov 03 20:47:34.647803 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.647776 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5"] Nov 03 20:47:34.647950 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.647932 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-6b87e780-dockercfg-jgvxq\"" Nov 03 20:47:34.661915 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.661880 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr"] Nov 03 20:47:34.665423 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.665401 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr" Nov 03 20:47:34.675109 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.675083 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr"] Nov 03 20:47:34.681966 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.681940 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15659b60-a0f6-4b5e-95e8-f4345378e19b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5\" (UID: \"15659b60-a0f6-4b5e-95e8-f4345378e19b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5" Nov 03 20:47:34.682084 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.682064 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjlc6\" (UniqueName: \"kubernetes.io/projected/15659b60-a0f6-4b5e-95e8-f4345378e19b-kube-api-access-xjlc6\") pod \"payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5\" (UID: \"15659b60-a0f6-4b5e-95e8-f4345378e19b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5" Nov 03 20:47:34.718852 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.718824 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-e95cf7\"" Nov 03 20:47:34.742402 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.742367 4918 generic.go:356] "Generic (PLEG): container finished" podID="da26a005-8c1e-46a4-8a17-93119a8a735d" containerID="ef21e567cda94fc66e98c4cfc1fc93a265404e8091928a8687dab1c20799410e" exitCode=0 Nov 03 20:47:34.742621 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.742456 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" event={"ID":"da26a005-8c1e-46a4-8a17-93119a8a735d","Type":"ContainerDied","Data":"ef21e567cda94fc66e98c4cfc1fc93a265404e8091928a8687dab1c20799410e"} Nov 03 20:47:34.742621 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.742532 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" event={"ID":"da26a005-8c1e-46a4-8a17-93119a8a735d","Type":"ContainerDied","Data":"80bed84f133bcda43ee467d76850e603878824cf2fc2c9011f0f34c371e2b10e"} Nov 03 20:47:34.742621 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.742570 4918 scope.go:122] "RemoveContainer" containerID="ef21e567cda94fc66e98c4cfc1fc93a265404e8091928a8687dab1c20799410e" Nov 03 20:47:34.742829 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.742474 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr" Nov 03 20:47:34.744923 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.744860 4918 generic.go:356] "Generic (PLEG): container finished" podID="67ba957b-0d42-473f-b2fb-c6c5fbea9885" containerID="2205fca82fe4e010a2e336d348e30c7a5fe774ed231504e4a292c2945f829ae9" exitCode=0 Nov 03 20:47:34.746546 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.744925 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" event={"ID":"67ba957b-0d42-473f-b2fb-c6c5fbea9885","Type":"ContainerDied","Data":"2205fca82fe4e010a2e336d348e30c7a5fe774ed231504e4a292c2945f829ae9"} Nov 03 20:47:34.746546 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.744965 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" event={"ID":"67ba957b-0d42-473f-b2fb-c6c5fbea9885","Type":"ContainerDied","Data":"e72dad55967f6fc55526715df191f34871d5a9804bf958b3ca0c7ec779fa7766"} Nov 03 20:47:34.746546 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.745036 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7" Nov 03 20:47:34.759710 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.759567 4918 scope.go:122] "RemoveContainer" containerID="ef21e567cda94fc66e98c4cfc1fc93a265404e8091928a8687dab1c20799410e" Nov 03 20:47:34.760165 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:47:34.760141 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef21e567cda94fc66e98c4cfc1fc93a265404e8091928a8687dab1c20799410e\": container with ID starting with ef21e567cda94fc66e98c4cfc1fc93a265404e8091928a8687dab1c20799410e not found: ID does not exist" containerID="ef21e567cda94fc66e98c4cfc1fc93a265404e8091928a8687dab1c20799410e" Nov 03 20:47:34.760287 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.760205 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef21e567cda94fc66e98c4cfc1fc93a265404e8091928a8687dab1c20799410e"} err="failed to get container status \"ef21e567cda94fc66e98c4cfc1fc93a265404e8091928a8687dab1c20799410e\": rpc error: code = NotFound desc = could not find container \"ef21e567cda94fc66e98c4cfc1fc93a265404e8091928a8687dab1c20799410e\": container with ID starting with ef21e567cda94fc66e98c4cfc1fc93a265404e8091928a8687dab1c20799410e not found: ID does not exist" Nov 03 20:47:34.760287 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.760230 4918 scope.go:122] "RemoveContainer" containerID="2205fca82fe4e010a2e336d348e30c7a5fe774ed231504e4a292c2945f829ae9" Nov 03 20:47:34.769948 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.769918 4918 scope.go:122] "RemoveContainer" containerID="2205fca82fe4e010a2e336d348e30c7a5fe774ed231504e4a292c2945f829ae9" Nov 03 20:47:34.770288 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:47:34.770268 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2205fca82fe4e010a2e336d348e30c7a5fe774ed231504e4a292c2945f829ae9\": container with ID starting with 2205fca82fe4e010a2e336d348e30c7a5fe774ed231504e4a292c2945f829ae9 not found: ID does not exist" containerID="2205fca82fe4e010a2e336d348e30c7a5fe774ed231504e4a292c2945f829ae9" Nov 03 20:47:34.770359 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.770299 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2205fca82fe4e010a2e336d348e30c7a5fe774ed231504e4a292c2945f829ae9"} err="failed to get container status \"2205fca82fe4e010a2e336d348e30c7a5fe774ed231504e4a292c2945f829ae9\": rpc error: code = NotFound desc = could not find container \"2205fca82fe4e010a2e336d348e30c7a5fe774ed231504e4a292c2945f829ae9\": container with ID starting with 2205fca82fe4e010a2e336d348e30c7a5fe774ed231504e4a292c2945f829ae9 not found: ID does not exist" Nov 03 20:47:34.774252 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.774221 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr"] Nov 03 20:47:34.779505 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.779481 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zjhwr"] Nov 03 20:47:34.782634 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.782611 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15659b60-a0f6-4b5e-95e8-f4345378e19b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5\" (UID: \"15659b60-a0f6-4b5e-95e8-f4345378e19b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5" Nov 03 20:47:34.782735 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.782662 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr\" (UID: \"ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr" Nov 03 20:47:34.782735 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.782704 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5rlh\" (UniqueName: \"kubernetes.io/projected/ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c-kube-api-access-p5rlh\") pod \"payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr\" (UID: \"ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr" Nov 03 20:47:34.782830 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.782740 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjlc6\" (UniqueName: \"kubernetes.io/projected/15659b60-a0f6-4b5e-95e8-f4345378e19b-kube-api-access-xjlc6\") pod \"payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5\" (UID: \"15659b60-a0f6-4b5e-95e8-f4345378e19b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5" Nov 03 20:47:34.783343 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.783323 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15659b60-a0f6-4b5e-95e8-f4345378e19b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5\" (UID: \"15659b60-a0f6-4b5e-95e8-f4345378e19b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5" Nov 03 20:47:34.790649 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.790622 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7"] Nov 03 20:47:34.794591 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.794565 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xwdt7"] Nov 03 20:47:34.796409 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.796388 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjlc6\" (UniqueName: \"kubernetes.io/projected/15659b60-a0f6-4b5e-95e8-f4345378e19b-kube-api-access-xjlc6\") pod \"payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5\" (UID: \"15659b60-a0f6-4b5e-95e8-f4345378e19b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5" Nov 03 20:47:34.884106 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.884017 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr\" (UID: \"ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr" Nov 03 20:47:34.884106 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.884071 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5rlh\" (UniqueName: \"kubernetes.io/projected/ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c-kube-api-access-p5rlh\") pod \"payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr\" (UID: \"ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr" Nov 03 20:47:34.884766 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.884742 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr\" (UID: \"ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr" Nov 03 20:47:34.896546 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.896513 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5rlh\" (UniqueName: \"kubernetes.io/projected/ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c-kube-api-access-p5rlh\") pod \"payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr\" (UID: \"ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr" Nov 03 20:47:34.948266 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.948228 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5" Nov 03 20:47:34.980918 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:34.980888 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr" Nov 03 20:47:35.213792 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:35.213705 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67ba957b-0d42-473f-b2fb-c6c5fbea9885" path="/var/lib/kubelet/pods/67ba957b-0d42-473f-b2fb-c6c5fbea9885/volumes" Nov 03 20:47:35.214385 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:35.214364 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da26a005-8c1e-46a4-8a17-93119a8a735d" path="/var/lib/kubelet/pods/da26a005-8c1e-46a4-8a17-93119a8a735d/volumes" Nov 03 20:47:35.231295 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:35.231272 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5"] Nov 03 20:47:35.233908 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:47:35.233877 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15659b60_a0f6_4b5e_95e8_f4345378e19b.slice/crio-a0b33312a51820282f5526fa9389c5ae332543f6ae6d4c3f1deb9039f7d7343b WatchSource:0}: Error finding container a0b33312a51820282f5526fa9389c5ae332543f6ae6d4c3f1deb9039f7d7343b: Status 404 returned error can't find the container with id a0b33312a51820282f5526fa9389c5ae332543f6ae6d4c3f1deb9039f7d7343b Nov 03 20:47:35.269692 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:35.269663 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr"] Nov 03 20:47:35.271375 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:47:35.271350 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7dc76b_85a5_47ee_81ae_c6ef90e6eb0c.slice/crio-26203c2280135a8604744c07ba5193c010823924ac9ea1081414fa4bd261b7ca WatchSource:0}: Error finding container 26203c2280135a8604744c07ba5193c010823924ac9ea1081414fa4bd261b7ca: Status 404 returned error can't find the container with id 26203c2280135a8604744c07ba5193c010823924ac9ea1081414fa4bd261b7ca Nov 03 20:47:35.751766 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:35.751727 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5" event={"ID":"15659b60-a0f6-4b5e-95e8-f4345378e19b","Type":"ContainerStarted","Data":"5bed76f3718da45ae34f3d11d80298a69a8bc46bb4ac268262d04fa5f467cb26"} Nov 03 20:47:35.751953 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:35.751787 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5" event={"ID":"15659b60-a0f6-4b5e-95e8-f4345378e19b","Type":"ContainerStarted","Data":"a0b33312a51820282f5526fa9389c5ae332543f6ae6d4c3f1deb9039f7d7343b"} Nov 03 20:47:35.751953 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:35.751838 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5" Nov 03 20:47:35.753231 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:35.753204 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr" event={"ID":"ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c","Type":"ContainerStarted","Data":"c1c4bc696f9440ea6953ac1f3eb81fbda05e6125e644c95a02d5235661a57599"} Nov 03 20:47:35.753393 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:35.753254 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr" Nov 03 20:47:35.753393 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:35.753271 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr" event={"ID":"ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c","Type":"ContainerStarted","Data":"26203c2280135a8604744c07ba5193c010823924ac9ea1081414fa4bd261b7ca"} Nov 03 20:47:35.771025 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:35.770978 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5" podStartSLOduration=1.7709662339999999 podStartE2EDuration="1.770966234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:35.76954725 +0000 UTC m=+3038.701414765" watchObservedRunningTime="2026-11-03 20:47:35.770966234 +0000 UTC m=+3038.702833745" Nov 03 20:47:35.787611 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:35.787511 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr" podStartSLOduration=1.787498102 podStartE2EDuration="1.787498102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:35.786831227 +0000 UTC m=+3038.718698740" watchObservedRunningTime="2026-11-03 20:47:35.787498102 +0000 UTC m=+3038.719365615" Nov 03 20:47:43.811057 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.811018 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq"] Nov 03 20:47:43.816599 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.816490 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.828496 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.828471 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq"] Nov 03 20:47:43.875165 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.875123 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.875360 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.875255 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.875360 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.875306 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.875360 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.875332 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.875472 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.875429 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.875551 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.875473 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.875551 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.875492 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s74rp\" (UniqueName: \"kubernetes.io/projected/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-kube-api-access-s74rp\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.875551 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.875526 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.875687 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.875672 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.976986 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.976951 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.976986 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.976989 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s74rp\" (UniqueName: \"kubernetes.io/projected/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-kube-api-access-s74rp\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.977290 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.977018 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.977290 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.977059 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.977290 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.977080 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.977290 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.977239 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.977515 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.977316 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.977515 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.977377 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.977515 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.977447 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.977515 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.977464 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.977719 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.977610 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.977719 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.977681 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.977839 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.977819 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.977899 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.977833 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.979451 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.979427 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.979973 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.979952 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.989228 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.989201 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:43.989429 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:43.989407 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s74rp\" (UniqueName: \"kubernetes.io/projected/7f1582e1-cf7e-448f-b9aa-73f1badd5c6b-kube-api-access-s74rp\") pod \"maas-default-gateway-openshift-default-77f556d647-nwfzq\" (UID: \"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:44.130707 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:44.130668 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:44.412190 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:44.412148 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq"] Nov 03 20:47:44.414278 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:47:44.414244 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f1582e1_cf7e_448f_b9aa_73f1badd5c6b.slice/crio-68a0fb820ab34af548a1b8ab2de9dfeabbfe8f5a3dfcd287bdedcca8182493aa WatchSource:0}: Error finding container 68a0fb820ab34af548a1b8ab2de9dfeabbfe8f5a3dfcd287bdedcca8182493aa: Status 404 returned error can't find the container with id 68a0fb820ab34af548a1b8ab2de9dfeabbfe8f5a3dfcd287bdedcca8182493aa Nov 03 20:47:44.416681 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:44.416638 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:47:44.416776 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:44.416739 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:47:44.416818 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:44.416783 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:47:44.804220 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:44.799650 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" event={"ID":"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b","Type":"ContainerStarted","Data":"c183095a3cb2262bc22c7973d821ab60cc1f722f5b2cd3232165d89b319b6576"} Nov 03 20:47:44.804220 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:44.799719 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" event={"ID":"7f1582e1-cf7e-448f-b9aa-73f1badd5c6b","Type":"ContainerStarted","Data":"68a0fb820ab34af548a1b8ab2de9dfeabbfe8f5a3dfcd287bdedcca8182493aa"} Nov 03 20:47:45.140202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:45.139581 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:45.143323 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:45.143287 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-nwfzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:47:45.143499 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:45.143358 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" podUID="7f1582e1-cf7e-448f-b9aa-73f1badd5c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 20:47:46.141204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:46.133627 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-nwfzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:47:46.141204 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:46.133712 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" podUID="7f1582e1-cf7e-448f-b9aa-73f1badd5c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 20:47:46.762932 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:46.762901 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr" Nov 03 20:47:46.763107 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:46.762967 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5" Nov 03 20:47:46.783235 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:46.783166 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" podStartSLOduration=3.783152267 podStartE2EDuration="3.783152267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:44.825872505 +0000 UTC m=+3047.757740017" watchObservedRunningTime="2026-11-03 20:47:46.783152267 +0000 UTC m=+3049.715019778" Nov 03 20:47:47.142202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:47.141646 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-nwfzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:47:47.142202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:47.141725 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" podUID="7f1582e1-cf7e-448f-b9aa-73f1badd5c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 20:47:48.138617 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:48.138572 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-nwfzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:47:48.138923 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:48.138893 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" podUID="7f1582e1-cf7e-448f-b9aa-73f1badd5c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 20:47:49.139201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:49.138562 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-nwfzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:47:49.139201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:49.138630 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" podUID="7f1582e1-cf7e-448f-b9aa-73f1badd5c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 20:47:50.138631 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:50.138592 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-nwfzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:47:50.138940 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:50.138911 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" podUID="7f1582e1-cf7e-448f-b9aa-73f1badd5c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 20:47:51.131357 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:51.131317 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-nwfzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:47:51.131804 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:51.131397 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" podUID="7f1582e1-cf7e-448f-b9aa-73f1badd5c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 20:47:52.669196 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:52.669146 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:52.669688 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:52.669546 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:52.670665 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:47:52.670640 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-nwfzq" Nov 03 20:47:58.213789 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:47:58.213681 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:48:28.208086 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:48:28.208039 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:48:41.863739 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:48:41.863644 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr"] Nov 03 20:48:41.873260 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:48:41.873224 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5"] Nov 03 20:48:46.864646 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:48:46.864590 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr" podUID="ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c" containerName="payload-processing" containerID="cri-o://c1c4bc696f9440ea6953ac1f3eb81fbda05e6125e644c95a02d5235661a57599" gracePeriod=25 Nov 03 20:48:46.874216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:48:46.874130 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5" podUID="15659b60-a0f6-4b5e-95e8-f4345378e19b" containerName="payload-pre-processing" containerID="cri-o://5bed76f3718da45ae34f3d11d80298a69a8bc46bb4ac268262d04fa5f467cb26" gracePeriod=25 Nov 03 20:48:56.762155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:48:56.762107 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5" podUID="15659b60-a0f6-4b5e-95e8-f4345378e19b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:48:56.762155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:48:56.762149 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr" podUID="ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:48:58.208101 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:48:58.208064 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:49:06.761891 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:06.761852 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr" podUID="ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:49:06.762411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:06.761849 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5" podUID="15659b60-a0f6-4b5e-95e8-f4345378e19b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:49:07.157715 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.157687 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr" Nov 03 20:49:07.189988 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.189956 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5" Nov 03 20:49:07.235931 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.235897 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c-kube-api-access-p5rlh\" (UniqueName: \"kubernetes.io/projected/ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c-kube-api-access-p5rlh\") pod \"ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c\" (UID: \"ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c\") " Nov 03 20:49:07.236120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.236010 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c-plugins-config-volume\") pod \"ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c\" (UID: \"ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c\") " Nov 03 20:49:07.236120 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.236075 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15659b60-a0f6-4b5e-95e8-f4345378e19b-kube-api-access-xjlc6\" (UniqueName: \"kubernetes.io/projected/15659b60-a0f6-4b5e-95e8-f4345378e19b-kube-api-access-xjlc6\") pod \"15659b60-a0f6-4b5e-95e8-f4345378e19b\" (UID: \"15659b60-a0f6-4b5e-95e8-f4345378e19b\") " Nov 03 20:49:07.236302 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.236123 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15659b60-a0f6-4b5e-95e8-f4345378e19b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15659b60-a0f6-4b5e-95e8-f4345378e19b-plugins-config-volume\") pod \"15659b60-a0f6-4b5e-95e8-f4345378e19b\" (UID: \"15659b60-a0f6-4b5e-95e8-f4345378e19b\") " Nov 03 20:49:07.236838 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.236812 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15659b60-a0f6-4b5e-95e8-f4345378e19b-plugins-config-volume" pod "15659b60-a0f6-4b5e-95e8-f4345378e19b" (UID: "15659b60-a0f6-4b5e-95e8-f4345378e19b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:49:07.237151 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.237127 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c-plugins-config-volume" pod "ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c" (UID: "ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:49:07.238156 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.238134 4918 generic.go:356] "Generic (PLEG): container finished" podID="15659b60-a0f6-4b5e-95e8-f4345378e19b" containerID="5bed76f3718da45ae34f3d11d80298a69a8bc46bb4ac268262d04fa5f467cb26" exitCode=0 Nov 03 20:49:07.238302 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.238245 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5" event={"ID":"15659b60-a0f6-4b5e-95e8-f4345378e19b","Type":"ContainerDied","Data":"5bed76f3718da45ae34f3d11d80298a69a8bc46bb4ac268262d04fa5f467cb26"} Nov 03 20:49:07.238302 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.238298 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5" event={"ID":"15659b60-a0f6-4b5e-95e8-f4345378e19b","Type":"ContainerDied","Data":"a0b33312a51820282f5526fa9389c5ae332543f6ae6d4c3f1deb9039f7d7343b"} Nov 03 20:49:07.238423 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.238328 4918 scope.go:122] "RemoveContainer" containerID="5bed76f3718da45ae34f3d11d80298a69a8bc46bb4ac268262d04fa5f467cb26" Nov 03 20:49:07.238492 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.238476 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5" Nov 03 20:49:07.239391 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.238997 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c-kube-api-access-p5rlh" pod "ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c" (UID: "ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c"). InnerVolumeSpecName "kube-api-access-p5rlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:49:07.239391 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.239010 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15659b60-a0f6-4b5e-95e8-f4345378e19b-kube-api-access-xjlc6" pod "15659b60-a0f6-4b5e-95e8-f4345378e19b" (UID: "15659b60-a0f6-4b5e-95e8-f4345378e19b"). InnerVolumeSpecName "kube-api-access-xjlc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:49:07.240529 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.240504 4918 generic.go:356] "Generic (PLEG): container finished" podID="ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c" containerID="c1c4bc696f9440ea6953ac1f3eb81fbda05e6125e644c95a02d5235661a57599" exitCode=0 Nov 03 20:49:07.240649 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.240556 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr" Nov 03 20:49:07.240649 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.240627 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr" event={"ID":"ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c","Type":"ContainerDied","Data":"c1c4bc696f9440ea6953ac1f3eb81fbda05e6125e644c95a02d5235661a57599"} Nov 03 20:49:07.240783 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.240708 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr" event={"ID":"ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c","Type":"ContainerDied","Data":"26203c2280135a8604744c07ba5193c010823924ac9ea1081414fa4bd261b7ca"} Nov 03 20:49:07.251050 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.251030 4918 scope.go:122] "RemoveContainer" containerID="5bed76f3718da45ae34f3d11d80298a69a8bc46bb4ac268262d04fa5f467cb26" Nov 03 20:49:07.251393 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:49:07.251366 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bed76f3718da45ae34f3d11d80298a69a8bc46bb4ac268262d04fa5f467cb26\": container with ID starting with 5bed76f3718da45ae34f3d11d80298a69a8bc46bb4ac268262d04fa5f467cb26 not found: ID does not exist" containerID="5bed76f3718da45ae34f3d11d80298a69a8bc46bb4ac268262d04fa5f467cb26" Nov 03 20:49:07.251492 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.251399 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bed76f3718da45ae34f3d11d80298a69a8bc46bb4ac268262d04fa5f467cb26"} err="failed to get container status \"5bed76f3718da45ae34f3d11d80298a69a8bc46bb4ac268262d04fa5f467cb26\": rpc error: code = NotFound desc = could not find container \"5bed76f3718da45ae34f3d11d80298a69a8bc46bb4ac268262d04fa5f467cb26\": container with ID starting with 5bed76f3718da45ae34f3d11d80298a69a8bc46bb4ac268262d04fa5f467cb26 not found: ID does not exist" Nov 03 20:49:07.251492 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.251418 4918 scope.go:122] "RemoveContainer" containerID="c1c4bc696f9440ea6953ac1f3eb81fbda05e6125e644c95a02d5235661a57599" Nov 03 20:49:07.260696 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.260674 4918 scope.go:122] "RemoveContainer" containerID="c1c4bc696f9440ea6953ac1f3eb81fbda05e6125e644c95a02d5235661a57599" Nov 03 20:49:07.261005 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:49:07.260983 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c4bc696f9440ea6953ac1f3eb81fbda05e6125e644c95a02d5235661a57599\": container with ID starting with c1c4bc696f9440ea6953ac1f3eb81fbda05e6125e644c95a02d5235661a57599 not found: ID does not exist" containerID="c1c4bc696f9440ea6953ac1f3eb81fbda05e6125e644c95a02d5235661a57599" Nov 03 20:49:07.261155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.261010 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c4bc696f9440ea6953ac1f3eb81fbda05e6125e644c95a02d5235661a57599"} err="failed to get container status \"c1c4bc696f9440ea6953ac1f3eb81fbda05e6125e644c95a02d5235661a57599\": rpc error: code = NotFound desc = could not find container \"c1c4bc696f9440ea6953ac1f3eb81fbda05e6125e644c95a02d5235661a57599\": container with ID starting with c1c4bc696f9440ea6953ac1f3eb81fbda05e6125e644c95a02d5235661a57599 not found: ID does not exist" Nov 03 20:49:07.266886 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.266864 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr"] Nov 03 20:49:07.270524 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.270498 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e95cf7-69d4d56b792dmr"] Nov 03 20:49:07.337631 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.337599 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5rlh\" (UniqueName: \"kubernetes.io/projected/ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c-kube-api-access-p5rlh\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:07.337631 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.337624 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:07.337631 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.337636 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjlc6\" (UniqueName: \"kubernetes.io/projected/15659b60-a0f6-4b5e-95e8-f4345378e19b-kube-api-access-xjlc6\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:07.337913 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.337646 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15659b60-a0f6-4b5e-95e8-f4345378e19b-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:07.562395 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.562356 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5"] Nov 03 20:49:07.567111 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:07.567079 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e95cf7-7588cmqwg5"] Nov 03 20:49:09.213651 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:09.213620 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15659b60-a0f6-4b5e-95e8-f4345378e19b" path="/var/lib/kubelet/pods/15659b60-a0f6-4b5e-95e8-f4345378e19b/volumes" Nov 03 20:49:09.214043 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:49:09.214023 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c" path="/var/lib/kubelet/pods/ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c/volumes" Nov 03 20:49:28.207794 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:49:28.207750 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:49:58.207220 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:49:58.207157 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:50:28.206897 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:50:28.206859 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:50:58.214724 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:50:58.214683 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:51:27.634524 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.634485 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw"] Nov 03 20:51:27.634983 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.634910 4918 cpu_manager.go:414] "containerMap: removing container" podUID="ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c" containerName="payload-processing" Nov 03 20:51:27.634983 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.634921 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c" containerName="payload-processing" Nov 03 20:51:27.634983 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.634956 4918 cpu_manager.go:414] "containerMap: removing container" podUID="15659b60-a0f6-4b5e-95e8-f4345378e19b" containerName="payload-pre-processing" Nov 03 20:51:27.634983 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.634961 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15659b60-a0f6-4b5e-95e8-f4345378e19b" containerName="payload-pre-processing" Nov 03 20:51:27.635140 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.635025 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="15659b60-a0f6-4b5e-95e8-f4345378e19b" containerName="payload-pre-processing" Nov 03 20:51:27.635140 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.635035 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab7dc76b-85a5-47ee-81ae-c6ef90e6eb0c" containerName="payload-processing" Nov 03 20:51:27.638272 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.638249 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.642102 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.642081 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-466ab7-openshift-default-dockercfg-qdq45\"" Nov 03 20:51:27.652427 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.652401 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw"] Nov 03 20:51:27.766566 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.766529 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw4qf\" (UniqueName: \"kubernetes.io/projected/5c40ce38-0691-44c0-aef0-55881b907be9-kube-api-access-dw4qf\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.766741 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.766583 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-istio-data\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.766741 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.766658 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c40ce38-0691-44c0-aef0-55881b907be9-istio-podinfo\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.766741 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.766713 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c40ce38-0691-44c0-aef0-55881b907be9-istio-token\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.766864 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.766782 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c40ce38-0691-44c0-aef0-55881b907be9-istiod-ca-cert\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.766864 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.766806 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-workload-certs\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.766864 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.766828 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-workload-socket\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.766864 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.766854 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-credential-socket\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.766998 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.766949 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-istio-envoy\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.868357 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.868316 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-istio-data\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.868357 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.868359 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c40ce38-0691-44c0-aef0-55881b907be9-istio-podinfo\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.868620 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.868397 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c40ce38-0691-44c0-aef0-55881b907be9-istio-token\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.868620 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.868431 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c40ce38-0691-44c0-aef0-55881b907be9-istiod-ca-cert\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.868620 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.868462 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-workload-certs\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.868620 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.868490 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-workload-socket\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.868620 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.868517 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-credential-socket\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.868620 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.868550 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-istio-envoy\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.868948 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.868643 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw4qf\" (UniqueName: \"kubernetes.io/projected/5c40ce38-0691-44c0-aef0-55881b907be9-kube-api-access-dw4qf\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.868948 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.868815 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-istio-data\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.868948 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.868906 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-workload-certs\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.869060 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.868950 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-workload-socket\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.869060 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.869002 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-credential-socket\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.869276 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.869246 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c40ce38-0691-44c0-aef0-55881b907be9-istiod-ca-cert\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.870959 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.870938 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-istio-envoy\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.871226 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.871206 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c40ce38-0691-44c0-aef0-55881b907be9-istio-podinfo\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.881220 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.881192 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c40ce38-0691-44c0-aef0-55881b907be9-istio-token\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.881365 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.881345 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw4qf\" (UniqueName: \"kubernetes.io/projected/5c40ce38-0691-44c0-aef0-55881b907be9-kube-api-access-dw4qf\") pod \"e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:27.952102 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:27.952027 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:28.207771 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:51:28.207662 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:51:28.234163 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:28.234139 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw"] Nov 03 20:51:28.236111 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:51:28.236069 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c40ce38_0691_44c0_aef0_55881b907be9.slice/crio-18f550037d5a50b324dd4c496bdc466588c93746a92dedf7c30241c71aa4d2e4 WatchSource:0}: Error finding container 18f550037d5a50b324dd4c496bdc466588c93746a92dedf7c30241c71aa4d2e4: Status 404 returned error can't find the container with id 18f550037d5a50b324dd4c496bdc466588c93746a92dedf7c30241c71aa4d2e4 Nov 03 20:51:28.238315 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:28.238284 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:51:28.238392 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:28.238380 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:51:28.238431 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:28.238408 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:51:28.932906 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:28.932866 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" event={"ID":"5c40ce38-0691-44c0-aef0-55881b907be9","Type":"ContainerStarted","Data":"5574323d823fd7778acdd40afbdc57f08e73287c1f9a3555a30348fd8d8df3b1"} Nov 03 20:51:28.933406 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:28.932924 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" event={"ID":"5c40ce38-0691-44c0-aef0-55881b907be9","Type":"ContainerStarted","Data":"18f550037d5a50b324dd4c496bdc466588c93746a92dedf7c30241c71aa4d2e4"} Nov 03 20:51:28.952224 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:28.952193 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:28.959202 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:28.959137 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" podStartSLOduration=1.959124893 podStartE2EDuration="1.959124893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:28.956231047 +0000 UTC m=+3271.888098559" watchObservedRunningTime="2026-11-03 20:51:28.959124893 +0000 UTC m=+3271.890992397" Nov 03 20:51:29.958530 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:29.958498 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:30.945529 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:30.945494 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:30.946646 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:30.946619 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:56.089972 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:56.089940 4918 generic.go:356] "Generic (PLEG): container finished" podID="5c40ce38-0691-44c0-aef0-55881b907be9" containerID="5574323d823fd7778acdd40afbdc57f08e73287c1f9a3555a30348fd8d8df3b1" exitCode=0 Nov 03 20:51:56.090499 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:56.090015 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" event={"ID":"5c40ce38-0691-44c0-aef0-55881b907be9","Type":"ContainerDied","Data":"5574323d823fd7778acdd40afbdc57f08e73287c1f9a3555a30348fd8d8df3b1"} Nov 03 20:51:56.090566 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:56.090548 4918 scope.go:122] "RemoveContainer" containerID="5574323d823fd7778acdd40afbdc57f08e73287c1f9a3555a30348fd8d8df3b1" Nov 03 20:51:56.091822 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:56.091794 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:51:56.091893 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:56.091861 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:51:56.091940 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:56.091894 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:51:57.097060 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:57.097023 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" event={"ID":"5c40ce38-0691-44c0-aef0-55881b907be9","Type":"ContainerStarted","Data":"612858a420e81a191841d0c5d25c462c779f3231c34ca9ba2d60173de20bf781"} Nov 03 20:51:57.666680 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:57.666651 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:51:57.678028 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:57.677999 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:51:57.953301 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:57.953201 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:51:57.954491 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:57.954454 4918 patch_prober.go:29] interesting pod/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 20:51:57.954667 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:57.954518 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" podUID="5c40ce38-0691-44c0-aef0-55881b907be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 20:51:58.209544 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:51:58.209461 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:51:58.953654 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:58.953605 4918 patch_prober.go:29] interesting pod/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 20:51:58.953879 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:58.953670 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" podUID="5c40ce38-0691-44c0-aef0-55881b907be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 20:51:59.952961 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:59.952921 4918 patch_prober.go:29] interesting pod/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 20:51:59.953502 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:51:59.952985 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" podUID="5c40ce38-0691-44c0-aef0-55881b907be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 20:52:00.946188 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:00.946138 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:52:00.952885 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:00.952857 4918 patch_prober.go:29] interesting pod/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 20:52:00.953030 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:00.952906 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" podUID="5c40ce38-0691-44c0-aef0-55881b907be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 20:52:01.953263 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:01.953223 4918 patch_prober.go:29] interesting pod/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 20:52:01.953746 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:01.953291 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" podUID="5c40ce38-0691-44c0-aef0-55881b907be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 20:52:02.957020 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:02.956983 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:52:02.957914 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:02.957887 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:52:28.208335 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:52:28.208279 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:52:54.595121 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:54.595082 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw"] Nov 03 20:52:54.595773 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:54.595455 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" podUID="5c40ce38-0691-44c0-aef0-55881b907be9" containerName="istio-proxy" containerID="cri-o://612858a420e81a191841d0c5d25c462c779f3231c34ca9ba2d60173de20bf781" gracePeriod=30 Nov 03 20:52:58.206873 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:52:58.206828 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:52:59.846272 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:59.846245 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:52:59.979780 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:59.979679 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-istio-data\") pod \"5c40ce38-0691-44c0-aef0-55881b907be9\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " Nov 03 20:52:59.979966 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:59.979790 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-workload-socket\") pod \"5c40ce38-0691-44c0-aef0-55881b907be9\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " Nov 03 20:52:59.979966 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:59.979863 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c40ce38-0691-44c0-aef0-55881b907be9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c40ce38-0691-44c0-aef0-55881b907be9-istiod-ca-cert\") pod \"5c40ce38-0691-44c0-aef0-55881b907be9\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " Nov 03 20:52:59.979966 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:59.979916 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-credential-socket\") pod \"5c40ce38-0691-44c0-aef0-55881b907be9\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " Nov 03 20:52:59.980162 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:59.979971 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-istio-envoy\") pod \"5c40ce38-0691-44c0-aef0-55881b907be9\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " Nov 03 20:52:59.980162 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:59.980007 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c40ce38-0691-44c0-aef0-55881b907be9-kube-api-access-dw4qf\" (UniqueName: \"kubernetes.io/projected/5c40ce38-0691-44c0-aef0-55881b907be9-kube-api-access-dw4qf\") pod \"5c40ce38-0691-44c0-aef0-55881b907be9\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " Nov 03 20:52:59.980162 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:59.980042 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c40ce38-0691-44c0-aef0-55881b907be9-istio-token\" (UniqueName: \"kubernetes.io/projected/5c40ce38-0691-44c0-aef0-55881b907be9-istio-token\") pod \"5c40ce38-0691-44c0-aef0-55881b907be9\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " Nov 03 20:52:59.980162 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:59.980054 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-workload-socket" pod "5c40ce38-0691-44c0-aef0-55881b907be9" (UID: "5c40ce38-0691-44c0-aef0-55881b907be9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:59.980162 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:59.980081 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-workload-certs\") pod \"5c40ce38-0691-44c0-aef0-55881b907be9\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " Nov 03 20:52:59.980162 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:59.980149 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c40ce38-0691-44c0-aef0-55881b907be9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c40ce38-0691-44c0-aef0-55881b907be9-istio-podinfo\") pod \"5c40ce38-0691-44c0-aef0-55881b907be9\" (UID: \"5c40ce38-0691-44c0-aef0-55881b907be9\") " Nov 03 20:52:59.980630 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:59.980605 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-workload-certs" pod "5c40ce38-0691-44c0-aef0-55881b907be9" (UID: "5c40ce38-0691-44c0-aef0-55881b907be9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:59.980735 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:59.980661 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:59.980929 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:59.980908 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c40ce38-0691-44c0-aef0-55881b907be9-istiod-ca-cert" pod "5c40ce38-0691-44c0-aef0-55881b907be9" (UID: "5c40ce38-0691-44c0-aef0-55881b907be9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:59.981247 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:59.981217 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-credential-socket" pod "5c40ce38-0691-44c0-aef0-55881b907be9" (UID: "5c40ce38-0691-44c0-aef0-55881b907be9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:59.981652 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:59.981629 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-istio-data" pod "5c40ce38-0691-44c0-aef0-55881b907be9" (UID: "5c40ce38-0691-44c0-aef0-55881b907be9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:59.985744 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:59.983033 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c40ce38-0691-44c0-aef0-55881b907be9-istio-token" pod "5c40ce38-0691-44c0-aef0-55881b907be9" (UID: "5c40ce38-0691-44c0-aef0-55881b907be9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:59.985744 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:59.983594 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c40ce38-0691-44c0-aef0-55881b907be9-istio-podinfo" pod "5c40ce38-0691-44c0-aef0-55881b907be9" (UID: "5c40ce38-0691-44c0-aef0-55881b907be9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:52:59.986506 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:59.986478 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c40ce38-0691-44c0-aef0-55881b907be9-kube-api-access-dw4qf" pod "5c40ce38-0691-44c0-aef0-55881b907be9" (UID: "5c40ce38-0691-44c0-aef0-55881b907be9"). InnerVolumeSpecName "kube-api-access-dw4qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:59.987074 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:52:59.987048 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-istio-envoy" pod "5c40ce38-0691-44c0-aef0-55881b907be9" (UID: "5c40ce38-0691-44c0-aef0-55881b907be9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:00.082129 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:00.082089 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c40ce38-0691-44c0-aef0-55881b907be9-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:00.082129 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:00.082131 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:00.082411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:00.082146 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c40ce38-0691-44c0-aef0-55881b907be9-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:00.082411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:00.082162 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:00.082411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:00.082203 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:00.082411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:00.082229 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw4qf\" (UniqueName: \"kubernetes.io/projected/5c40ce38-0691-44c0-aef0-55881b907be9-kube-api-access-dw4qf\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:00.082411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:00.082260 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c40ce38-0691-44c0-aef0-55881b907be9-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:00.082411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:00.082274 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c40ce38-0691-44c0-aef0-55881b907be9-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:00.409987 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:00.409939 4918 generic.go:356] "Generic (PLEG): container finished" podID="5c40ce38-0691-44c0-aef0-55881b907be9" containerID="612858a420e81a191841d0c5d25c462c779f3231c34ca9ba2d60173de20bf781" exitCode=0 Nov 03 20:53:00.410386 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:00.410341 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" event={"ID":"5c40ce38-0691-44c0-aef0-55881b907be9","Type":"ContainerDied","Data":"612858a420e81a191841d0c5d25c462c779f3231c34ca9ba2d60173de20bf781"} Nov 03 20:53:00.410609 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:00.410575 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" event={"ID":"5c40ce38-0691-44c0-aef0-55881b907be9","Type":"ContainerDied","Data":"18f550037d5a50b324dd4c496bdc466588c93746a92dedf7c30241c71aa4d2e4"} Nov 03 20:53:00.410773 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:00.410749 4918 scope.go:122] "RemoveContainer" containerID="612858a420e81a191841d0c5d25c462c779f3231c34ca9ba2d60173de20bf781" Nov 03 20:53:00.411121 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:00.411087 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw" Nov 03 20:53:00.443040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:00.442924 4918 scope.go:122] "RemoveContainer" containerID="5574323d823fd7778acdd40afbdc57f08e73287c1f9a3555a30348fd8d8df3b1" Nov 03 20:53:00.461820 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:00.461419 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw"] Nov 03 20:53:00.462790 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:00.462578 4918 scope.go:122] "RemoveContainer" containerID="612858a420e81a191841d0c5d25c462c779f3231c34ca9ba2d60173de20bf781" Nov 03 20:53:00.462982 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:53:00.462935 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"612858a420e81a191841d0c5d25c462c779f3231c34ca9ba2d60173de20bf781\": container with ID starting with 612858a420e81a191841d0c5d25c462c779f3231c34ca9ba2d60173de20bf781 not found: ID does not exist" containerID="612858a420e81a191841d0c5d25c462c779f3231c34ca9ba2d60173de20bf781" Nov 03 20:53:00.463095 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:00.462971 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"612858a420e81a191841d0c5d25c462c779f3231c34ca9ba2d60173de20bf781"} err="failed to get container status \"612858a420e81a191841d0c5d25c462c779f3231c34ca9ba2d60173de20bf781\": rpc error: code = NotFound desc = could not find container \"612858a420e81a191841d0c5d25c462c779f3231c34ca9ba2d60173de20bf781\": container with ID starting with 612858a420e81a191841d0c5d25c462c779f3231c34ca9ba2d60173de20bf781 not found: ID does not exist" Nov 03 20:53:00.463095 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:00.463012 4918 scope.go:122] "RemoveContainer" containerID="5574323d823fd7778acdd40afbdc57f08e73287c1f9a3555a30348fd8d8df3b1" Nov 03 20:53:00.463411 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:53:00.463365 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5574323d823fd7778acdd40afbdc57f08e73287c1f9a3555a30348fd8d8df3b1\": container with ID starting with 5574323d823fd7778acdd40afbdc57f08e73287c1f9a3555a30348fd8d8df3b1 not found: ID does not exist" containerID="5574323d823fd7778acdd40afbdc57f08e73287c1f9a3555a30348fd8d8df3b1" Nov 03 20:53:00.463546 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:00.463508 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5574323d823fd7778acdd40afbdc57f08e73287c1f9a3555a30348fd8d8df3b1"} err="failed to get container status \"5574323d823fd7778acdd40afbdc57f08e73287c1f9a3555a30348fd8d8df3b1\": rpc error: code = NotFound desc = could not find container \"5574323d823fd7778acdd40afbdc57f08e73287c1f9a3555a30348fd8d8df3b1\": container with ID starting with 5574323d823fd7778acdd40afbdc57f08e73287c1f9a3555a30348fd8d8df3b1 not found: ID does not exist" Nov 03 20:53:00.466001 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:00.465979 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-466ab7-openshift-default-6d7575c745ks4qw"] Nov 03 20:53:01.214773 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:01.214716 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c40ce38-0691-44c0-aef0-55881b907be9" path="/var/lib/kubelet/pods/5c40ce38-0691-44c0-aef0-55881b907be9/volumes" Nov 03 20:53:15.373931 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.373893 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs"] Nov 03 20:53:15.374514 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.374312 4918 cpu_manager.go:414] "containerMap: removing container" podUID="5c40ce38-0691-44c0-aef0-55881b907be9" containerName="istio-proxy" Nov 03 20:53:15.374514 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.374324 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c40ce38-0691-44c0-aef0-55881b907be9" containerName="istio-proxy" Nov 03 20:53:15.374514 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.374346 4918 cpu_manager.go:414] "containerMap: removing container" podUID="5c40ce38-0691-44c0-aef0-55881b907be9" containerName="istio-proxy" Nov 03 20:53:15.374514 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.374351 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c40ce38-0691-44c0-aef0-55881b907be9" containerName="istio-proxy" Nov 03 20:53:15.374514 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.374419 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c40ce38-0691-44c0-aef0-55881b907be9" containerName="istio-proxy" Nov 03 20:53:15.378958 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.378930 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:15.387880 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.387855 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-ab04f05f-kserve-self-signed-certs\"" Nov 03 20:53:15.390773 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.390743 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs"] Nov 03 20:53:15.555302 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.555261 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-tmp-dir\") pod \"e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:15.555460 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.555314 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b48lh\" (UniqueName: \"kubernetes.io/projected/983b2c69-7345-458b-ad79-3c0a59da1554-kube-api-access-b48lh\") pod \"e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:15.555460 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.555401 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983b2c69-7345-458b-ad79-3c0a59da1554-tls-certs\") pod \"e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:15.555460 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.555445 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-dshm\") pod \"e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:15.555623 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.555479 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-model-cache\") pod \"e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:15.555623 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.555598 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-home\") pod \"e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:15.656898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.656808 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-model-cache\") pod \"e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:15.656898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.656872 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-home\") pod \"e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:15.657097 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.656978 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-tmp-dir\") pod \"e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:15.657097 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.657018 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b48lh\" (UniqueName: \"kubernetes.io/projected/983b2c69-7345-458b-ad79-3c0a59da1554-kube-api-access-b48lh\") pod \"e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:15.657097 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.657047 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983b2c69-7345-458b-ad79-3c0a59da1554-tls-certs\") pod \"e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:15.657097 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.657091 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-dshm\") pod \"e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:15.657349 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.657267 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-model-cache\") pod \"e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:15.657349 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.657331 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-home\") pod \"e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:15.657456 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.657399 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-tmp-dir\") pod \"e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:15.659517 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.659488 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-dshm\") pod \"e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:15.659631 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.659531 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983b2c69-7345-458b-ad79-3c0a59da1554-tls-certs\") pod \"e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:15.669805 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.669781 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b48lh\" (UniqueName: \"kubernetes.io/projected/983b2c69-7345-458b-ad79-3c0a59da1554-kube-api-access-b48lh\") pod \"e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:15.691598 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.691575 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:15.969211 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:15.969117 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs"] Nov 03 20:53:15.972924 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:53:15.972894 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod983b2c69_7345_458b_ad79_3c0a59da1554.slice/crio-cc52177a89c4150a8ca9e15c36d27067ef0f0ca0d1dc5354c502cb7b1a443a0d WatchSource:0}: Error finding container cc52177a89c4150a8ca9e15c36d27067ef0f0ca0d1dc5354c502cb7b1a443a0d: Status 404 returned error can't find the container with id cc52177a89c4150a8ca9e15c36d27067ef0f0ca0d1dc5354c502cb7b1a443a0d Nov 03 20:53:16.516246 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:16.516205 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" event={"ID":"983b2c69-7345-458b-ad79-3c0a59da1554","Type":"ContainerStarted","Data":"98eaa4dd54f581867d3751f1f8247efd0f270468880c1510d40233e0ef5fde3c"} Nov 03 20:53:16.516648 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:16.516255 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" event={"ID":"983b2c69-7345-458b-ad79-3c0a59da1554","Type":"ContainerStarted","Data":"cc52177a89c4150a8ca9e15c36d27067ef0f0ca0d1dc5354c502cb7b1a443a0d"} Nov 03 20:53:16.535788 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:16.535714 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" podStartSLOduration=1.5356976119999999 podStartE2EDuration="1.535697612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:16.534498199 +0000 UTC m=+3379.466365712" watchObservedRunningTime="2026-11-03 20:53:16.535697612 +0000 UTC m=+3379.467565123" Nov 03 20:53:16.692248 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:16.692213 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:19.943201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:19.941261 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8bvn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:53:19.943201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:19.941340 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" podUID="ea819fe0-461b-4952-a40e-7a47e8f51580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:25.692467 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:25.692418 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:25.705195 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:25.705159 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:26.591442 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:26.591410 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:28.206981 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:53:28.206939 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:53:43.806391 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:43.806337 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs"] Nov 03 20:53:49.926413 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:49.926367 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8bvn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:53:49.926942 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:49.926466 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" podUID="ea819fe0-461b-4952-a40e-7a47e8f51580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:58.208166 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:53:58.208130 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:53:58.831659 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:58.831617 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" podUID="983b2c69-7345-458b-ad79-3c0a59da1554" containerName="main" containerID="cri-o://98eaa4dd54f581867d3751f1f8247efd0f270468880c1510d40233e0ef5fde3c" gracePeriod=45 Nov 03 20:53:59.136292 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.136249 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:59.185744 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.185712 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-model-cache\") pod \"983b2c69-7345-458b-ad79-3c0a59da1554\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " Nov 03 20:53:59.185956 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.185791 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-dshm\" (UniqueName: \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-dshm\") pod \"983b2c69-7345-458b-ad79-3c0a59da1554\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " Nov 03 20:53:59.185956 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.185880 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/983b2c69-7345-458b-ad79-3c0a59da1554-kube-api-access-b48lh\" (UniqueName: \"kubernetes.io/projected/983b2c69-7345-458b-ad79-3c0a59da1554-kube-api-access-b48lh\") pod \"983b2c69-7345-458b-ad79-3c0a59da1554\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " Nov 03 20:53:59.185956 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.185914 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/983b2c69-7345-458b-ad79-3c0a59da1554-tls-certs\" (UniqueName: \"kubernetes.io/secret/983b2c69-7345-458b-ad79-3c0a59da1554-tls-certs\") pod \"983b2c69-7345-458b-ad79-3c0a59da1554\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " Nov 03 20:53:59.185956 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.185944 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-tmp-dir\") pod \"983b2c69-7345-458b-ad79-3c0a59da1554\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " Nov 03 20:53:59.186311 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.185976 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-home\" (UniqueName: \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-home\") pod \"983b2c69-7345-458b-ad79-3c0a59da1554\" (UID: \"983b2c69-7345-458b-ad79-3c0a59da1554\") " Nov 03 20:53:59.186553 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.186532 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-home" pod "983b2c69-7345-458b-ad79-3c0a59da1554" (UID: "983b2c69-7345-458b-ad79-3c0a59da1554"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:59.186701 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.186671 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-model-cache" pod "983b2c69-7345-458b-ad79-3c0a59da1554" (UID: "983b2c69-7345-458b-ad79-3c0a59da1554"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:59.186928 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.186901 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-tmp-dir" pod "983b2c69-7345-458b-ad79-3c0a59da1554" (UID: "983b2c69-7345-458b-ad79-3c0a59da1554"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:59.188776 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.188751 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/983b2c69-7345-458b-ad79-3c0a59da1554-tls-certs" pod "983b2c69-7345-458b-ad79-3c0a59da1554" (UID: "983b2c69-7345-458b-ad79-3c0a59da1554"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:53:59.189689 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.189652 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-dshm" pod "983b2c69-7345-458b-ad79-3c0a59da1554" (UID: "983b2c69-7345-458b-ad79-3c0a59da1554"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:59.191028 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.190991 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983b2c69-7345-458b-ad79-3c0a59da1554-kube-api-access-b48lh" pod "983b2c69-7345-458b-ad79-3c0a59da1554" (UID: "983b2c69-7345-458b-ad79-3c0a59da1554"). InnerVolumeSpecName "kube-api-access-b48lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:53:59.289197 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.288836 4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-dshm\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:59.289197 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.288901 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b48lh\" (UniqueName: \"kubernetes.io/projected/983b2c69-7345-458b-ad79-3c0a59da1554-kube-api-access-b48lh\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:59.289197 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.288920 4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983b2c69-7345-458b-ad79-3c0a59da1554-tls-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:59.289197 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.288935 4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-tmp-dir\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:59.289197 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.288949 4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-home\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:59.289197 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.288963 4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/983b2c69-7345-458b-ad79-3c0a59da1554-model-cache\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:59.807257 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.807099 4918 generic.go:356] "Generic (PLEG): container finished" podID="983b2c69-7345-458b-ad79-3c0a59da1554" containerID="98eaa4dd54f581867d3751f1f8247efd0f270468880c1510d40233e0ef5fde3c" exitCode=0 Nov 03 20:53:59.807257 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.807206 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" event={"ID":"983b2c69-7345-458b-ad79-3c0a59da1554","Type":"ContainerDied","Data":"98eaa4dd54f581867d3751f1f8247efd0f270468880c1510d40233e0ef5fde3c"} Nov 03 20:53:59.807257 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.807246 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" event={"ID":"983b2c69-7345-458b-ad79-3c0a59da1554","Type":"ContainerDied","Data":"cc52177a89c4150a8ca9e15c36d27067ef0f0ca0d1dc5354c502cb7b1a443a0d"} Nov 03 20:53:59.807543 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.807269 4918 scope.go:122] "RemoveContainer" containerID="98eaa4dd54f581867d3751f1f8247efd0f270468880c1510d40233e0ef5fde3c" Nov 03 20:53:59.807543 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.807297 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs" Nov 03 20:53:59.817059 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.817037 4918 scope.go:122] "RemoveContainer" containerID="98eaa4dd54f581867d3751f1f8247efd0f270468880c1510d40233e0ef5fde3c" Nov 03 20:53:59.817364 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:53:59.817334 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98eaa4dd54f581867d3751f1f8247efd0f270468880c1510d40233e0ef5fde3c\": container with ID starting with 98eaa4dd54f581867d3751f1f8247efd0f270468880c1510d40233e0ef5fde3c not found: ID does not exist" containerID="98eaa4dd54f581867d3751f1f8247efd0f270468880c1510d40233e0ef5fde3c" Nov 03 20:53:59.817436 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.817366 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98eaa4dd54f581867d3751f1f8247efd0f270468880c1510d40233e0ef5fde3c"} err="failed to get container status \"98eaa4dd54f581867d3751f1f8247efd0f270468880c1510d40233e0ef5fde3c\": rpc error: code = NotFound desc = could not find container \"98eaa4dd54f581867d3751f1f8247efd0f270468880c1510d40233e0ef5fde3c\": container with ID starting with 98eaa4dd54f581867d3751f1f8247efd0f270468880c1510d40233e0ef5fde3c not found: ID does not exist" Nov 03 20:53:59.855288 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.855242 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs"] Nov 03 20:53:59.862203 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:53:59.857445 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-ab04f05f-kserve-6c6fd9b6b5-d8prs"] Nov 03 20:54:01.212950 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:01.212908 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="983b2c69-7345-458b-ad79-3c0a59da1554" path="/var/lib/kubelet/pods/983b2c69-7345-458b-ad79-3c0a59da1554/volumes" Nov 03 20:54:28.207710 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:54:28.207667 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:54:36.447936 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.447897 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2"] Nov 03 20:54:36.448476 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.448322 4918 cpu_manager.go:414] "containerMap: removing container" podUID="983b2c69-7345-458b-ad79-3c0a59da1554" containerName="main" Nov 03 20:54:36.448476 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.448334 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="983b2c69-7345-458b-ad79-3c0a59da1554" containerName="main" Nov 03 20:54:36.448476 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.448419 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="983b2c69-7345-458b-ad79-3c0a59da1554" containerName="main" Nov 03 20:54:36.448476 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.448436 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c40ce38-0691-44c0-aef0-55881b907be9" containerName="istio-proxy" Nov 03 20:54:36.450907 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.450880 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.456639 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.456588 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bdf98686-openshift-default-dockercfg-7qgr7\"" Nov 03 20:54:36.466940 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.466909 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2"] Nov 03 20:54:36.552085 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.552053 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-workload-certs\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.552286 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.552097 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-podinfo\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.552286 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.552133 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af827a46-c0d1-4161-836f-7a487c67e5f7-istiod-ca-cert\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.552381 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.552290 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8klcj\" (UniqueName: \"kubernetes.io/projected/af827a46-c0d1-4161-836f-7a487c67e5f7-kube-api-access-8klcj\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.552381 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.552343 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-data\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.552485 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.552391 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-credential-socket\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.552547 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.552499 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-envoy\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.552607 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.552570 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-workload-socket\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.552706 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.552681 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-token\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.653394 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.653355 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-workload-certs\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.653394 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.653399 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-podinfo\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.653667 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.653522 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af827a46-c0d1-4161-836f-7a487c67e5f7-istiod-ca-cert\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.653667 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.653605 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8klcj\" (UniqueName: \"kubernetes.io/projected/af827a46-c0d1-4161-836f-7a487c67e5f7-kube-api-access-8klcj\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.653667 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.653638 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-data\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.653831 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.653693 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-credential-socket\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.653831 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.653740 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-envoy\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.653831 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.653790 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-workload-certs\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.653831 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.653819 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-workload-socket\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.654044 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.653870 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-token\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.654103 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.654065 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-data\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.654163 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.654099 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-credential-socket\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.654261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.654246 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af827a46-c0d1-4161-836f-7a487c67e5f7-istiod-ca-cert\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.654326 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.654244 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-workload-socket\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.656002 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.655973 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-podinfo\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.656151 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.656128 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-envoy\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.665882 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.665859 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8klcj\" (UniqueName: \"kubernetes.io/projected/af827a46-c0d1-4161-836f-7a487c67e5f7-kube-api-access-8klcj\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.666000 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.665940 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-token\") pod \"e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:36.764389 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:36.764306 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:37.051250 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:37.051219 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2"] Nov 03 20:54:37.053254 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:54:37.053232 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf827a46_c0d1_4161_836f_7a487c67e5f7.slice/crio-d8e9d99ff35499445694ed4ef296f4d64dbbc186a6397a26f4f8dc22f4da8302 WatchSource:0}: Error finding container d8e9d99ff35499445694ed4ef296f4d64dbbc186a6397a26f4f8dc22f4da8302: Status 404 returned error can't find the container with id d8e9d99ff35499445694ed4ef296f4d64dbbc186a6397a26f4f8dc22f4da8302 Nov 03 20:54:37.055673 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:37.055640 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:54:37.055766 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:37.055704 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:54:37.055766 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:37.055734 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:54:37.985903 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:37.985815 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" event={"ID":"af827a46-c0d1-4161-836f-7a487c67e5f7","Type":"ContainerStarted","Data":"91633fb5118451936a9d374f2f754ded4ddec8a4d410ccfd1b955a25aac48f29"} Nov 03 20:54:37.985903 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:37.985899 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" event={"ID":"af827a46-c0d1-4161-836f-7a487c67e5f7","Type":"ContainerStarted","Data":"d8e9d99ff35499445694ed4ef296f4d64dbbc186a6397a26f4f8dc22f4da8302"} Nov 03 20:54:38.011167 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:38.011119 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" podStartSLOduration=2.011106994 podStartE2EDuration="2.011106994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:38.010335891 +0000 UTC m=+3460.942203402" watchObservedRunningTime="2026-11-03 20:54:38.011106994 +0000 UTC m=+3460.942974505" Nov 03 20:54:38.765478 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:38.765443 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:38.770508 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:38.770480 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:38.990705 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:38.990666 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:38.991448 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:38.991426 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:40.898262 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:40.898232 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk"] Nov 03 20:54:40.902318 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:40.902296 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk" Nov 03 20:54:40.936201 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:40.936150 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-bdf98686\"" Nov 03 20:54:40.937138 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:40.937119 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-bdf98686-dockercfg-7hhd9\"" Nov 03 20:54:40.959053 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:40.959001 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk"] Nov 03 20:54:40.997348 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:40.997314 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4mbs\" (UniqueName: \"kubernetes.io/projected/d22ffc86-9850-411b-bc5a-14ddad38cdc8-kube-api-access-v4mbs\") pod \"payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk\" (UID: \"d22ffc86-9850-411b-bc5a-14ddad38cdc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk" Nov 03 20:54:40.997491 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:40.997373 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22ffc86-9850-411b-bc5a-14ddad38cdc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk\" (UID: \"d22ffc86-9850-411b-bc5a-14ddad38cdc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk" Nov 03 20:54:41.007779 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:41.007755 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv"] Nov 03 20:54:41.011379 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:41.011355 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv" Nov 03 20:54:41.055345 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:41.055311 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv"] Nov 03 20:54:41.098540 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:41.098503 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26vrw\" (UniqueName: \"kubernetes.io/projected/eb3cfacb-a533-4a40-a242-3d3b9afff996-kube-api-access-26vrw\") pod \"payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv\" (UID: \"eb3cfacb-a533-4a40-a242-3d3b9afff996\") " pod="openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv" Nov 03 20:54:41.098762 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:41.098650 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4mbs\" (UniqueName: \"kubernetes.io/projected/d22ffc86-9850-411b-bc5a-14ddad38cdc8-kube-api-access-v4mbs\") pod \"payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk\" (UID: \"d22ffc86-9850-411b-bc5a-14ddad38cdc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk" Nov 03 20:54:41.098762 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:41.098724 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb3cfacb-a533-4a40-a242-3d3b9afff996-plugins-config-volume\") pod \"payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv\" (UID: \"eb3cfacb-a533-4a40-a242-3d3b9afff996\") " pod="openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv" Nov 03 20:54:41.098894 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:41.098810 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22ffc86-9850-411b-bc5a-14ddad38cdc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk\" (UID: \"d22ffc86-9850-411b-bc5a-14ddad38cdc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk" Nov 03 20:54:41.099690 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:41.099667 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22ffc86-9850-411b-bc5a-14ddad38cdc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk\" (UID: \"d22ffc86-9850-411b-bc5a-14ddad38cdc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk" Nov 03 20:54:41.111944 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:41.111908 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4mbs\" (UniqueName: \"kubernetes.io/projected/d22ffc86-9850-411b-bc5a-14ddad38cdc8-kube-api-access-v4mbs\") pod \"payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk\" (UID: \"d22ffc86-9850-411b-bc5a-14ddad38cdc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk" Nov 03 20:54:41.200470 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:41.200375 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26vrw\" (UniqueName: \"kubernetes.io/projected/eb3cfacb-a533-4a40-a242-3d3b9afff996-kube-api-access-26vrw\") pod \"payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv\" (UID: \"eb3cfacb-a533-4a40-a242-3d3b9afff996\") " pod="openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv" Nov 03 20:54:41.200635 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:41.200482 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb3cfacb-a533-4a40-a242-3d3b9afff996-plugins-config-volume\") pod \"payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv\" (UID: \"eb3cfacb-a533-4a40-a242-3d3b9afff996\") " pod="openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv" Nov 03 20:54:41.201206 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:41.201155 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb3cfacb-a533-4a40-a242-3d3b9afff996-plugins-config-volume\") pod \"payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv\" (UID: \"eb3cfacb-a533-4a40-a242-3d3b9afff996\") " pod="openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv" Nov 03 20:54:41.212772 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:41.212733 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26vrw\" (UniqueName: \"kubernetes.io/projected/eb3cfacb-a533-4a40-a242-3d3b9afff996-kube-api-access-26vrw\") pod \"payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv\" (UID: \"eb3cfacb-a533-4a40-a242-3d3b9afff996\") " pod="openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv" Nov 03 20:54:41.215287 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:41.215265 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk" Nov 03 20:54:41.323991 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:41.323958 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv" Nov 03 20:54:41.490675 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:41.490315 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk"] Nov 03 20:54:41.496627 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:54:41.496582 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22ffc86_9850_411b_bc5a_14ddad38cdc8.slice/crio-1dcea3211e5fca06260450d47fffed9188032140cc1b0734baee2858f805804a WatchSource:0}: Error finding container 1dcea3211e5fca06260450d47fffed9188032140cc1b0734baee2858f805804a: Status 404 returned error can't find the container with id 1dcea3211e5fca06260450d47fffed9188032140cc1b0734baee2858f805804a Nov 03 20:54:41.606831 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:41.606803 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv"] Nov 03 20:54:41.609219 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:54:41.609189 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3cfacb_a533_4a40_a242_3d3b9afff996.slice/crio-6134dbedb9a4e1be9563bec7a7d1ee4d70d31b06369288113f4013ec0ee551b9 WatchSource:0}: Error finding container 6134dbedb9a4e1be9563bec7a7d1ee4d70d31b06369288113f4013ec0ee551b9: Status 404 returned error can't find the container with id 6134dbedb9a4e1be9563bec7a7d1ee4d70d31b06369288113f4013ec0ee551b9 Nov 03 20:54:42.005402 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:42.005363 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk" event={"ID":"d22ffc86-9850-411b-bc5a-14ddad38cdc8","Type":"ContainerStarted","Data":"55317f8caae0350f223e90dd8d7a859bd9ef0942dcb10e392db3d20772b71baf"} Nov 03 20:54:42.005867 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:42.005430 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk" event={"ID":"d22ffc86-9850-411b-bc5a-14ddad38cdc8","Type":"ContainerStarted","Data":"1dcea3211e5fca06260450d47fffed9188032140cc1b0734baee2858f805804a"} Nov 03 20:54:42.005867 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:42.005546 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk" Nov 03 20:54:42.007083 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:42.007057 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv" event={"ID":"eb3cfacb-a533-4a40-a242-3d3b9afff996","Type":"ContainerStarted","Data":"d8122690818c6e677dff880391bd762dc0e47bb0daff08d661bc3f64d3d451cf"} Nov 03 20:54:42.007231 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:42.007107 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv" event={"ID":"eb3cfacb-a533-4a40-a242-3d3b9afff996","Type":"ContainerStarted","Data":"6134dbedb9a4e1be9563bec7a7d1ee4d70d31b06369288113f4013ec0ee551b9"} Nov 03 20:54:42.007231 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:42.007159 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv" Nov 03 20:54:42.023530 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:42.023485 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk" podStartSLOduration=2.023471636 podStartE2EDuration="2.023471636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:42.022619469 +0000 UTC m=+3464.954486983" watchObservedRunningTime="2026-11-03 20:54:42.023471636 +0000 UTC m=+3464.955339148" Nov 03 20:54:42.041554 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:42.041501 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv" podStartSLOduration=2.041484926 podStartE2EDuration="2.041484926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:42.040038786 +0000 UTC m=+3464.971906298" watchObservedRunningTime="2026-11-03 20:54:42.041484926 +0000 UTC m=+3464.973352438" Nov 03 20:54:44.790652 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:44.790615 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p"] Nov 03 20:54:44.791068 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:44.790873 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" podUID="3bb27585-623d-46bb-ae29-046c214e2252" containerName="maas-api" containerID="cri-o://23911820db778d4a8d1f046223ca32315a286935adf68d857414417346d6c301" gracePeriod=30 Nov 03 20:54:46.056951 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:46.056917 4918 generic.go:356] "Generic (PLEG): container finished" podID="3bb27585-623d-46bb-ae29-046c214e2252" containerID="23911820db778d4a8d1f046223ca32315a286935adf68d857414417346d6c301" exitCode=0 Nov 03 20:54:46.057462 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:46.056991 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" event={"ID":"3bb27585-623d-46bb-ae29-046c214e2252","Type":"ContainerDied","Data":"23911820db778d4a8d1f046223ca32315a286935adf68d857414417346d6c301"} Nov 03 20:54:46.100503 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:46.100443 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" Nov 03 20:54:46.168303 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:46.168267 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bb27585-623d-46bb-ae29-046c214e2252-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bb27585-623d-46bb-ae29-046c214e2252-metrics-certs\") pod \"3bb27585-623d-46bb-ae29-046c214e2252\" (UID: \"3bb27585-623d-46bb-ae29-046c214e2252\") " Nov 03 20:54:46.168501 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:46.168322 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb27585-623d-46bb-ae29-046c214e2252-kube-api-access-shprg\" (UniqueName: \"kubernetes.io/projected/3bb27585-623d-46bb-ae29-046c214e2252-kube-api-access-shprg\") pod \"3bb27585-623d-46bb-ae29-046c214e2252\" (UID: \"3bb27585-623d-46bb-ae29-046c214e2252\") " Nov 03 20:54:46.168501 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:46.168381 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bb27585-623d-46bb-ae29-046c214e2252-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bb27585-623d-46bb-ae29-046c214e2252-openshift-service-ca\") pod \"3bb27585-623d-46bb-ae29-046c214e2252\" (UID: \"3bb27585-623d-46bb-ae29-046c214e2252\") " Nov 03 20:54:46.168501 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:46.168463 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bb27585-623d-46bb-ae29-046c214e2252-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bb27585-623d-46bb-ae29-046c214e2252-maas-api-tls\") pod \"3bb27585-623d-46bb-ae29-046c214e2252\" (UID: \"3bb27585-623d-46bb-ae29-046c214e2252\") " Nov 03 20:54:46.168833 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:46.168804 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb27585-623d-46bb-ae29-046c214e2252-openshift-service-ca" pod "3bb27585-623d-46bb-ae29-046c214e2252" (UID: "3bb27585-623d-46bb-ae29-046c214e2252"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:54:46.170588 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:46.170557 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bb27585-623d-46bb-ae29-046c214e2252-metrics-certs" pod "3bb27585-623d-46bb-ae29-046c214e2252" (UID: "3bb27585-623d-46bb-ae29-046c214e2252"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:46.170701 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:46.170603 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bb27585-623d-46bb-ae29-046c214e2252-maas-api-tls" pod "3bb27585-623d-46bb-ae29-046c214e2252" (UID: "3bb27585-623d-46bb-ae29-046c214e2252"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:46.170701 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:46.170661 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb27585-623d-46bb-ae29-046c214e2252-kube-api-access-shprg" pod "3bb27585-623d-46bb-ae29-046c214e2252" (UID: "3bb27585-623d-46bb-ae29-046c214e2252"). InnerVolumeSpecName "kube-api-access-shprg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:46.270129 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:46.270090 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bb27585-623d-46bb-ae29-046c214e2252-maas-api-tls\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:46.270129 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:46.270121 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bb27585-623d-46bb-ae29-046c214e2252-metrics-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:46.270129 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:46.270131 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shprg\" (UniqueName: \"kubernetes.io/projected/3bb27585-623d-46bb-ae29-046c214e2252-kube-api-access-shprg\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:46.270129 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:46.270140 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bb27585-623d-46bb-ae29-046c214e2252-openshift-service-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:47.062648 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:47.062617 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" Nov 03 20:54:47.063142 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:47.062656 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p" event={"ID":"3bb27585-623d-46bb-ae29-046c214e2252","Type":"ContainerDied","Data":"62955e577e1ae41a45963fbafcbcd8a002431ef20ff5dd24bea720aec173829e"} Nov 03 20:54:47.063142 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:47.062731 4918 scope.go:122] "RemoveContainer" containerID="23911820db778d4a8d1f046223ca32315a286935adf68d857414417346d6c301" Nov 03 20:54:47.087953 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:47.087927 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p"] Nov 03 20:54:47.091996 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:47.091972 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c684ff7f7-qrf9p"] Nov 03 20:54:47.214339 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:47.214305 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bb27585-623d-46bb-ae29-046c214e2252" path="/var/lib/kubelet/pods/3bb27585-623d-46bb-ae29-046c214e2252/volumes" Nov 03 20:54:53.024144 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:53.016612 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv" Nov 03 20:54:53.024144 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:53.017293 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk" Nov 03 20:54:53.095485 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:53.095454 4918 generic.go:356] "Generic (PLEG): container finished" podID="af827a46-c0d1-4161-836f-7a487c67e5f7" containerID="91633fb5118451936a9d374f2f754ded4ddec8a4d410ccfd1b955a25aac48f29" exitCode=0 Nov 03 20:54:53.095655 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:53.095522 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" event={"ID":"af827a46-c0d1-4161-836f-7a487c67e5f7","Type":"ContainerDied","Data":"91633fb5118451936a9d374f2f754ded4ddec8a4d410ccfd1b955a25aac48f29"} Nov 03 20:54:53.095939 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:53.095923 4918 scope.go:122] "RemoveContainer" containerID="91633fb5118451936a9d374f2f754ded4ddec8a4d410ccfd1b955a25aac48f29" Nov 03 20:54:53.097236 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:53.097208 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:54:53.097310 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:53.097296 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:54:53.097351 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:53.097328 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:54:53.991411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:53.991379 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:54.102084 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:54.102047 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" event={"ID":"af827a46-c0d1-4161-836f-7a487c67e5f7","Type":"ContainerStarted","Data":"767048c1e9c91e3aa25c547422c936b6b9fc9b898a554f2047f61ec660beae88"} Nov 03 20:54:54.764740 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:54.764701 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:54.765957 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:54.765928 4918 patch_prober.go:29] interesting pod/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 20:54:54.766089 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:54.765980 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" podUID="af827a46-c0d1-4161-836f-7a487c67e5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 20:54:55.403937 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:55.403903 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt"] Nov 03 20:54:55.404381 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:55.404365 4918 cpu_manager.go:414] "containerMap: removing container" podUID="3bb27585-623d-46bb-ae29-046c214e2252" containerName="maas-api" Nov 03 20:54:55.404425 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:55.404382 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb27585-623d-46bb-ae29-046c214e2252" containerName="maas-api" Nov 03 20:54:55.404470 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:55.404461 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb27585-623d-46bb-ae29-046c214e2252" containerName="maas-api" Nov 03 20:54:55.407443 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:55.407423 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" Nov 03 20:54:55.415310 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:55.415290 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:54:55.415417 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:55.415372 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:54:55.415417 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:55.415385 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:54:55.415534 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:55.415371 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:54:55.427199 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:55.418258 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt"] Nov 03 20:54:55.461531 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:55.461495 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf203d3d-a52b-4919-adab-3a48d1f301de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt\" (UID: \"bf203d3d-a52b-4919-adab-3a48d1f301de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" Nov 03 20:54:55.461693 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:55.461536 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf203d3d-a52b-4919-adab-3a48d1f301de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt\" (UID: \"bf203d3d-a52b-4919-adab-3a48d1f301de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" Nov 03 20:54:55.461693 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:55.461652 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvh4b\" (UniqueName: \"kubernetes.io/projected/bf203d3d-a52b-4919-adab-3a48d1f301de-kube-api-access-xvh4b\") pod \"maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt\" (UID: \"bf203d3d-a52b-4919-adab-3a48d1f301de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" Nov 03 20:54:55.563224 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:55.563155 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf203d3d-a52b-4919-adab-3a48d1f301de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt\" (UID: \"bf203d3d-a52b-4919-adab-3a48d1f301de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" Nov 03 20:54:55.563407 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:55.563248 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf203d3d-a52b-4919-adab-3a48d1f301de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt\" (UID: \"bf203d3d-a52b-4919-adab-3a48d1f301de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" Nov 03 20:54:55.563407 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:55.563351 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvh4b\" (UniqueName: \"kubernetes.io/projected/bf203d3d-a52b-4919-adab-3a48d1f301de-kube-api-access-xvh4b\") pod \"maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt\" (UID: \"bf203d3d-a52b-4919-adab-3a48d1f301de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" Nov 03 20:54:55.563866 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:55.563845 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf203d3d-a52b-4919-adab-3a48d1f301de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt\" (UID: \"bf203d3d-a52b-4919-adab-3a48d1f301de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" Nov 03 20:54:55.565775 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:55.565757 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf203d3d-a52b-4919-adab-3a48d1f301de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt\" (UID: \"bf203d3d-a52b-4919-adab-3a48d1f301de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" Nov 03 20:54:55.569749 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:54:55.569731 4918 projected.go:196] Error preparing data for projected volume kube-api-access-xvh4b for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 03 20:54:55.569807 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:54:55.569799 4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf203d3d-a52b-4919-adab-3a48d1f301de-kube-api-access-xvh4b podName:bf203d3d-a52b-4919-adab-3a48d1f301de nodeName:}" failed. No retries permitted until 2026-11-03 20:54:56.069777288 +0000 UTC m=+3479.001644777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xvh4b" (UniqueName: "kubernetes.io/projected/bf203d3d-a52b-4919-adab-3a48d1f301de-kube-api-access-xvh4b") pod "maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" (UID: "bf203d3d-a52b-4919-adab-3a48d1f301de") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 03 20:54:55.765487 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:55.765399 4918 patch_prober.go:29] interesting pod/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 20:54:55.765487 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:55.765459 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" podUID="af827a46-c0d1-4161-836f-7a487c67e5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 20:54:56.169793 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:56.169740 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvh4b\" (UniqueName: \"kubernetes.io/projected/bf203d3d-a52b-4919-adab-3a48d1f301de-kube-api-access-xvh4b\") pod \"maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt\" (UID: \"bf203d3d-a52b-4919-adab-3a48d1f301de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" Nov 03 20:54:56.182442 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:56.182413 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvh4b\" (UniqueName: \"kubernetes.io/projected/bf203d3d-a52b-4919-adab-3a48d1f301de-kube-api-access-xvh4b\") pod \"maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt\" (UID: \"bf203d3d-a52b-4919-adab-3a48d1f301de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" Nov 03 20:54:56.343369 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:56.343326 4918 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" secret="" err="secret \"maas-api-cleanup-dockercfg-xzp5p\" not found" Nov 03 20:54:56.343549 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:56.343391 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" Nov 03 20:54:56.645471 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:56.619569 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt"] Nov 03 20:54:56.647431 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:54:56.647393 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf203d3d_a52b_4919_adab_3a48d1f301de.slice/crio-3cc50ab065b768b3cdeb23bb21c73bb8bd7ab5a38ebd37f9a17fa01ad9ee3e90 WatchSource:0}: Error finding container 3cc50ab065b768b3cdeb23bb21c73bb8bd7ab5a38ebd37f9a17fa01ad9ee3e90: Status 404 returned error can't find the container with id 3cc50ab065b768b3cdeb23bb21c73bb8bd7ab5a38ebd37f9a17fa01ad9ee3e90 Nov 03 20:54:56.765103 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:56.765056 4918 patch_prober.go:29] interesting pod/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 20:54:56.765300 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:56.765130 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" podUID="af827a46-c0d1-4161-836f-7a487c67e5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 20:54:57.118897 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:57.118855 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" event={"ID":"bf203d3d-a52b-4919-adab-3a48d1f301de","Type":"ContainerStarted","Data":"b72174e4db0fe9303d0cd9683c45f3800d2eb8938794060813d7bc2dff705af3"} Nov 03 20:54:57.132313 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:57.132271 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" event={"ID":"bf203d3d-a52b-4919-adab-3a48d1f301de","Type":"ContainerStarted","Data":"3cc50ab065b768b3cdeb23bb21c73bb8bd7ab5a38ebd37f9a17fa01ad9ee3e90"} Nov 03 20:54:57.132571 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:57.119555 4918 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" secret="" err="secret \"maas-api-cleanup-dockercfg-xzp5p\" not found" Nov 03 20:54:57.155944 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:57.155888 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" podStartSLOduration=2.155874683 podStartE2EDuration="2.155874683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:57.15472071 +0000 UTC m=+3480.086588246" watchObservedRunningTime="2026-11-03 20:54:57.155874683 +0000 UTC m=+3480.087742191" Nov 03 20:54:57.765040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:57.765001 4918 patch_prober.go:29] interesting pod/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 20:54:57.765486 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:57.765063 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" podUID="af827a46-c0d1-4161-836f-7a487c67e5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 20:54:58.154173 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:58.154132 4918 generic.go:356] "Generic (PLEG): container finished" podID="bf203d3d-a52b-4919-adab-3a48d1f301de" containerID="b72174e4db0fe9303d0cd9683c45f3800d2eb8938794060813d7bc2dff705af3" exitCode=0 Nov 03 20:54:58.154398 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:58.154215 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" event={"ID":"bf203d3d-a52b-4919-adab-3a48d1f301de","Type":"ContainerDied","Data":"b72174e4db0fe9303d0cd9683c45f3800d2eb8938794060813d7bc2dff705af3"} Nov 03 20:54:58.209167 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:54:58.209112 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:54:58.765708 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:58.765674 4918 patch_prober.go:29] interesting pod/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 20:54:58.766123 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:58.765727 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" podUID="af827a46-c0d1-4161-836f-7a487c67e5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 20:54:59.297854 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:59.297822 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" Nov 03 20:54:59.401935 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:59.401893 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf203d3d-a52b-4919-adab-3a48d1f301de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf203d3d-a52b-4919-adab-3a48d1f301de-maas-api-openshift-ca\") pod \"bf203d3d-a52b-4919-adab-3a48d1f301de\" (UID: \"bf203d3d-a52b-4919-adab-3a48d1f301de\") " Nov 03 20:54:59.402113 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:59.401970 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf203d3d-a52b-4919-adab-3a48d1f301de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf203d3d-a52b-4919-adab-3a48d1f301de-maas-api-xks-ca\") pod \"bf203d3d-a52b-4919-adab-3a48d1f301de\" (UID: \"bf203d3d-a52b-4919-adab-3a48d1f301de\") " Nov 03 20:54:59.402113 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:59.402001 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf203d3d-a52b-4919-adab-3a48d1f301de-kube-api-access-xvh4b\" (UniqueName: \"kubernetes.io/projected/bf203d3d-a52b-4919-adab-3a48d1f301de-kube-api-access-xvh4b\") pod \"bf203d3d-a52b-4919-adab-3a48d1f301de\" (UID: \"bf203d3d-a52b-4919-adab-3a48d1f301de\") " Nov 03 20:54:59.402307 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:59.402282 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf203d3d-a52b-4919-adab-3a48d1f301de-maas-api-openshift-ca" pod "bf203d3d-a52b-4919-adab-3a48d1f301de" (UID: "bf203d3d-a52b-4919-adab-3a48d1f301de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:54:59.404258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:59.404234 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf203d3d-a52b-4919-adab-3a48d1f301de-maas-api-xks-ca" pod "bf203d3d-a52b-4919-adab-3a48d1f301de" (UID: "bf203d3d-a52b-4919-adab-3a48d1f301de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:59.404345 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:59.404256 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf203d3d-a52b-4919-adab-3a48d1f301de-kube-api-access-xvh4b" pod "bf203d3d-a52b-4919-adab-3a48d1f301de" (UID: "bf203d3d-a52b-4919-adab-3a48d1f301de"). InnerVolumeSpecName "kube-api-access-xvh4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:59.503812 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:59.503771 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf203d3d-a52b-4919-adab-3a48d1f301de-maas-api-openshift-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:59.503812 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:59.503805 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf203d3d-a52b-4919-adab-3a48d1f301de-maas-api-xks-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:59.503812 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:59.503815 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvh4b\" (UniqueName: \"kubernetes.io/projected/bf203d3d-a52b-4919-adab-3a48d1f301de-kube-api-access-xvh4b\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:59.768935 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:59.768851 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:59.769313 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:59.769167 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:54:59.769902 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:54:59.769882 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:55:00.165532 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:00.165492 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" event={"ID":"bf203d3d-a52b-4919-adab-3a48d1f301de","Type":"ContainerDied","Data":"3cc50ab065b768b3cdeb23bb21c73bb8bd7ab5a38ebd37f9a17fa01ad9ee3e90"} Nov 03 20:55:00.165715 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:00.165555 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cc50ab065b768b3cdeb23bb21c73bb8bd7ab5a38ebd37f9a17fa01ad9ee3e90" Nov 03 20:55:00.165715 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:00.165509 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt" Nov 03 20:55:05.549449 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:05.549414 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv"] Nov 03 20:55:05.564170 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:05.564139 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk"] Nov 03 20:55:06.499320 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:06.499283 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt"] Nov 03 20:55:06.500967 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:06.500942 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bdf98686-4w5nt"] Nov 03 20:55:07.213265 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:07.213229 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf203d3d-a52b-4919-adab-3a48d1f301de" path="/var/lib/kubelet/pods/bf203d3d-a52b-4919-adab-3a48d1f301de/volumes" Nov 03 20:55:10.549754 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:10.549662 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv" podUID="eb3cfacb-a533-4a40-a242-3d3b9afff996" containerName="payload-processing" containerID="cri-o://d8122690818c6e677dff880391bd762dc0e47bb0daff08d661bc3f64d3d451cf" gracePeriod=25 Nov 03 20:55:10.564688 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:10.564647 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk" podUID="d22ffc86-9850-411b-bc5a-14ddad38cdc8" containerName="payload-pre-processing" containerID="cri-o://55317f8caae0350f223e90dd8d7a859bd9ef0942dcb10e392db3d20772b71baf" gracePeriod=25 Nov 03 20:55:10.816011 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:10.815983 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv" Nov 03 20:55:10.847812 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:10.847782 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk" Nov 03 20:55:10.916527 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:10.916491 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb3cfacb-a533-4a40-a242-3d3b9afff996-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb3cfacb-a533-4a40-a242-3d3b9afff996-plugins-config-volume\") pod \"eb3cfacb-a533-4a40-a242-3d3b9afff996\" (UID: \"eb3cfacb-a533-4a40-a242-3d3b9afff996\") " Nov 03 20:55:10.916729 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:10.916579 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d22ffc86-9850-411b-bc5a-14ddad38cdc8-kube-api-access-v4mbs\" (UniqueName: \"kubernetes.io/projected/d22ffc86-9850-411b-bc5a-14ddad38cdc8-kube-api-access-v4mbs\") pod \"d22ffc86-9850-411b-bc5a-14ddad38cdc8\" (UID: \"d22ffc86-9850-411b-bc5a-14ddad38cdc8\") " Nov 03 20:55:10.916729 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:10.916614 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb3cfacb-a533-4a40-a242-3d3b9afff996-kube-api-access-26vrw\" (UniqueName: \"kubernetes.io/projected/eb3cfacb-a533-4a40-a242-3d3b9afff996-kube-api-access-26vrw\") pod \"eb3cfacb-a533-4a40-a242-3d3b9afff996\" (UID: \"eb3cfacb-a533-4a40-a242-3d3b9afff996\") " Nov 03 20:55:10.916729 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:10.916670 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d22ffc86-9850-411b-bc5a-14ddad38cdc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22ffc86-9850-411b-bc5a-14ddad38cdc8-plugins-config-volume\") pod \"d22ffc86-9850-411b-bc5a-14ddad38cdc8\" (UID: \"d22ffc86-9850-411b-bc5a-14ddad38cdc8\") " Nov 03 20:55:10.916925 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:10.916882 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb3cfacb-a533-4a40-a242-3d3b9afff996-plugins-config-volume" pod "eb3cfacb-a533-4a40-a242-3d3b9afff996" (UID: "eb3cfacb-a533-4a40-a242-3d3b9afff996"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:10.917156 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:10.917122 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22ffc86-9850-411b-bc5a-14ddad38cdc8-plugins-config-volume" pod "d22ffc86-9850-411b-bc5a-14ddad38cdc8" (UID: "d22ffc86-9850-411b-bc5a-14ddad38cdc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:10.918783 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:10.918758 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3cfacb-a533-4a40-a242-3d3b9afff996-kube-api-access-26vrw" pod "eb3cfacb-a533-4a40-a242-3d3b9afff996" (UID: "eb3cfacb-a533-4a40-a242-3d3b9afff996"). InnerVolumeSpecName "kube-api-access-26vrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:10.918901 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:10.918776 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22ffc86-9850-411b-bc5a-14ddad38cdc8-kube-api-access-v4mbs" pod "d22ffc86-9850-411b-bc5a-14ddad38cdc8" (UID: "d22ffc86-9850-411b-bc5a-14ddad38cdc8"). InnerVolumeSpecName "kube-api-access-v4mbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:11.018046 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.018012 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb3cfacb-a533-4a40-a242-3d3b9afff996-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:11.018046 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.018041 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4mbs\" (UniqueName: \"kubernetes.io/projected/d22ffc86-9850-411b-bc5a-14ddad38cdc8-kube-api-access-v4mbs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:11.018046 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.018050 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26vrw\" (UniqueName: \"kubernetes.io/projected/eb3cfacb-a533-4a40-a242-3d3b9afff996-kube-api-access-26vrw\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:11.018315 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.018060 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22ffc86-9850-411b-bc5a-14ddad38cdc8-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:11.221686 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.221595 4918 generic.go:356] "Generic (PLEG): container finished" podID="d22ffc86-9850-411b-bc5a-14ddad38cdc8" containerID="55317f8caae0350f223e90dd8d7a859bd9ef0942dcb10e392db3d20772b71baf" exitCode=0 Nov 03 20:55:11.221686 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.221632 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk" Nov 03 20:55:11.221917 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.221681 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk" event={"ID":"d22ffc86-9850-411b-bc5a-14ddad38cdc8","Type":"ContainerDied","Data":"55317f8caae0350f223e90dd8d7a859bd9ef0942dcb10e392db3d20772b71baf"} Nov 03 20:55:11.221917 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.221726 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk" event={"ID":"d22ffc86-9850-411b-bc5a-14ddad38cdc8","Type":"ContainerDied","Data":"1dcea3211e5fca06260450d47fffed9188032140cc1b0734baee2858f805804a"} Nov 03 20:55:11.221917 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.221755 4918 scope.go:122] "RemoveContainer" containerID="55317f8caae0350f223e90dd8d7a859bd9ef0942dcb10e392db3d20772b71baf" Nov 03 20:55:11.223337 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.223311 4918 generic.go:356] "Generic (PLEG): container finished" podID="eb3cfacb-a533-4a40-a242-3d3b9afff996" containerID="d8122690818c6e677dff880391bd762dc0e47bb0daff08d661bc3f64d3d451cf" exitCode=0 Nov 03 20:55:11.223447 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.223353 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv" Nov 03 20:55:11.223447 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.223364 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv" event={"ID":"eb3cfacb-a533-4a40-a242-3d3b9afff996","Type":"ContainerDied","Data":"d8122690818c6e677dff880391bd762dc0e47bb0daff08d661bc3f64d3d451cf"} Nov 03 20:55:11.223447 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.223398 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv" event={"ID":"eb3cfacb-a533-4a40-a242-3d3b9afff996","Type":"ContainerDied","Data":"6134dbedb9a4e1be9563bec7a7d1ee4d70d31b06369288113f4013ec0ee551b9"} Nov 03 20:55:11.231333 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.231318 4918 scope.go:122] "RemoveContainer" containerID="55317f8caae0350f223e90dd8d7a859bd9ef0942dcb10e392db3d20772b71baf" Nov 03 20:55:11.231573 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:55:11.231556 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55317f8caae0350f223e90dd8d7a859bd9ef0942dcb10e392db3d20772b71baf\": container with ID starting with 55317f8caae0350f223e90dd8d7a859bd9ef0942dcb10e392db3d20772b71baf not found: ID does not exist" containerID="55317f8caae0350f223e90dd8d7a859bd9ef0942dcb10e392db3d20772b71baf" Nov 03 20:55:11.231625 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.231581 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55317f8caae0350f223e90dd8d7a859bd9ef0942dcb10e392db3d20772b71baf"} err="failed to get container status \"55317f8caae0350f223e90dd8d7a859bd9ef0942dcb10e392db3d20772b71baf\": rpc error: code = NotFound desc = could not find container \"55317f8caae0350f223e90dd8d7a859bd9ef0942dcb10e392db3d20772b71baf\": container with ID starting with 55317f8caae0350f223e90dd8d7a859bd9ef0942dcb10e392db3d20772b71baf not found: ID does not exist" Nov 03 20:55:11.231625 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.231596 4918 scope.go:122] "RemoveContainer" containerID="d8122690818c6e677dff880391bd762dc0e47bb0daff08d661bc3f64d3d451cf" Nov 03 20:55:11.239058 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.239042 4918 scope.go:122] "RemoveContainer" containerID="d8122690818c6e677dff880391bd762dc0e47bb0daff08d661bc3f64d3d451cf" Nov 03 20:55:11.239317 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:55:11.239300 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8122690818c6e677dff880391bd762dc0e47bb0daff08d661bc3f64d3d451cf\": container with ID starting with d8122690818c6e677dff880391bd762dc0e47bb0daff08d661bc3f64d3d451cf not found: ID does not exist" containerID="d8122690818c6e677dff880391bd762dc0e47bb0daff08d661bc3f64d3d451cf" Nov 03 20:55:11.239381 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.239322 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8122690818c6e677dff880391bd762dc0e47bb0daff08d661bc3f64d3d451cf"} err="failed to get container status \"d8122690818c6e677dff880391bd762dc0e47bb0daff08d661bc3f64d3d451cf\": rpc error: code = NotFound desc = could not find container \"d8122690818c6e677dff880391bd762dc0e47bb0daff08d661bc3f64d3d451cf\": container with ID starting with d8122690818c6e677dff880391bd762dc0e47bb0daff08d661bc3f64d3d451cf not found: ID does not exist" Nov 03 20:55:11.242486 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.242465 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv"] Nov 03 20:55:11.246762 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.246734 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bdf98686-67fb999b4d-xjqgv"] Nov 03 20:55:11.258367 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.258349 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk"] Nov 03 20:55:11.262476 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.262455 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bdf98686-66cb9b6547-m5rsk"] Nov 03 20:55:11.824235 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.824119 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2"] Nov 03 20:55:11.824780 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:11.824404 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" podUID="af827a46-c0d1-4161-836f-7a487c67e5f7" containerName="istio-proxy" containerID="cri-o://767048c1e9c91e3aa25c547422c936b6b9fc9b898a554f2047f61ec660beae88" gracePeriod=30 Nov 03 20:55:13.214168 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.214120 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d22ffc86-9850-411b-bc5a-14ddad38cdc8" path="/var/lib/kubelet/pods/d22ffc86-9850-411b-bc5a-14ddad38cdc8/volumes" Nov 03 20:55:13.214756 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.214725 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb3cfacb-a533-4a40-a242-3d3b9afff996" path="/var/lib/kubelet/pods/eb3cfacb-a533-4a40-a242-3d3b9afff996/volumes" Nov 03 20:55:13.527812 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.527711 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng"] Nov 03 20:55:13.528116 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.528099 4918 cpu_manager.go:414] "containerMap: removing container" podUID="d22ffc86-9850-411b-bc5a-14ddad38cdc8" containerName="payload-pre-processing" Nov 03 20:55:13.528116 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.528112 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d22ffc86-9850-411b-bc5a-14ddad38cdc8" containerName="payload-pre-processing" Nov 03 20:55:13.528351 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.528136 4918 cpu_manager.go:414] "containerMap: removing container" podUID="bf203d3d-a52b-4919-adab-3a48d1f301de" containerName="revoke-keys" Nov 03 20:55:13.528351 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.528141 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf203d3d-a52b-4919-adab-3a48d1f301de" containerName="revoke-keys" Nov 03 20:55:13.528351 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.528148 4918 cpu_manager.go:414] "containerMap: removing container" podUID="eb3cfacb-a533-4a40-a242-3d3b9afff996" containerName="payload-processing" Nov 03 20:55:13.528351 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.528153 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb3cfacb-a533-4a40-a242-3d3b9afff996" containerName="payload-processing" Nov 03 20:55:13.528351 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.528247 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d22ffc86-9850-411b-bc5a-14ddad38cdc8" containerName="payload-pre-processing" Nov 03 20:55:13.528351 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.528261 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf203d3d-a52b-4919-adab-3a48d1f301de" containerName="revoke-keys" Nov 03 20:55:13.528351 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.528267 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb3cfacb-a533-4a40-a242-3d3b9afff996" containerName="payload-processing" Nov 03 20:55:13.532907 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.532881 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.539296 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.539274 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-0fe875-openshift-default-dockercfg-hrj9t\"" Nov 03 20:55:13.546251 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.546229 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng"] Nov 03 20:55:13.645494 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.645452 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-workload-certs\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.645836 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.645594 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-envoy\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.645836 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.645637 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-token\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.645979 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.645826 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-workload-socket\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.646092 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.646052 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-data\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.646758 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.646104 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69rsp\" (UniqueName: \"kubernetes.io/projected/5c71bc62-125e-4573-8370-cbb43ac7677e-kube-api-access-69rsp\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.646758 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.646325 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-podinfo\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.646758 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.646365 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c71bc62-125e-4573-8370-cbb43ac7677e-istiod-ca-cert\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.646758 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.646463 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-credential-socket\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.747169 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.747129 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-credential-socket\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.747169 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.747200 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-workload-certs\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.747525 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.747237 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-envoy\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.747525 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.747263 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-token\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.747525 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.747330 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-workload-socket\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.747525 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.747399 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-data\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.747525 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.747427 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69rsp\" (UniqueName: \"kubernetes.io/projected/5c71bc62-125e-4573-8370-cbb43ac7677e-kube-api-access-69rsp\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.747525 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.747457 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-podinfo\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.747525 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.747480 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c71bc62-125e-4573-8370-cbb43ac7677e-istiod-ca-cert\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.747851 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.747674 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-credential-socket\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.747851 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.747711 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-workload-certs\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.747851 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.747840 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-workload-socket\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.747967 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.747881 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-data\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.748261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.748240 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c71bc62-125e-4573-8370-cbb43ac7677e-istiod-ca-cert\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.749892 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.749872 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-envoy\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.750000 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.749982 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-podinfo\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.759782 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.759748 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-token\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.759878 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.759822 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69rsp\" (UniqueName: \"kubernetes.io/projected/5c71bc62-125e-4573-8370-cbb43ac7677e-kube-api-access-69rsp\") pod \"e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:13.847056 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:13.847015 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:14.128415 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:14.128387 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng"] Nov 03 20:55:14.131132 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:55:14.131108 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c71bc62_125e_4573_8370_cbb43ac7677e.slice/crio-eb8616494079227f2c4dec2a5687e291f8e3a2de2cb1d8e70d6d8a5f0e478e7f WatchSource:0}: Error finding container eb8616494079227f2c4dec2a5687e291f8e3a2de2cb1d8e70d6d8a5f0e478e7f: Status 404 returned error can't find the container with id eb8616494079227f2c4dec2a5687e291f8e3a2de2cb1d8e70d6d8a5f0e478e7f Nov 03 20:55:14.133255 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:14.133222 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:55:14.133317 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:14.133287 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:55:14.133357 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:14.133314 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:55:14.241848 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:14.241809 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" event={"ID":"5c71bc62-125e-4573-8370-cbb43ac7677e","Type":"ContainerStarted","Data":"ef365bd01bb643fed333a0774be8f0b2ccd3589bdcc1996ed64970d3151dfaac"} Nov 03 20:55:14.242351 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:14.241871 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" event={"ID":"5c71bc62-125e-4573-8370-cbb43ac7677e","Type":"ContainerStarted","Data":"eb8616494079227f2c4dec2a5687e291f8e3a2de2cb1d8e70d6d8a5f0e478e7f"} Nov 03 20:55:14.267154 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:14.267103 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" podStartSLOduration=1.267088095 podStartE2EDuration="1.267088095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:14.266168355 +0000 UTC m=+3497.198035867" watchObservedRunningTime="2026-11-03 20:55:14.267088095 +0000 UTC m=+3497.198955626" Nov 03 20:55:14.771662 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:14.771632 4918 patch_prober.go:29] interesting pod/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:55:14.771820 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:14.771684 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" podUID="af827a46-c0d1-4161-836f-7a487c67e5f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:55:14.847690 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:14.847652 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:15.853079 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:15.853046 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:16.251373 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:16.251291 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:16.252211 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:16.252189 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:17.086843 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.086816 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:55:17.183788 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.183705 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-workload-socket\") pod \"af827a46-c0d1-4161-836f-7a487c67e5f7\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " Nov 03 20:55:17.183788 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.183748 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-envoy\") pod \"af827a46-c0d1-4161-836f-7a487c67e5f7\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " Nov 03 20:55:17.183788 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.183773 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-workload-certs\") pod \"af827a46-c0d1-4161-836f-7a487c67e5f7\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " Nov 03 20:55:17.184056 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.183799 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-podinfo\") pod \"af827a46-c0d1-4161-836f-7a487c67e5f7\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " Nov 03 20:55:17.184056 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.183842 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-data\") pod \"af827a46-c0d1-4161-836f-7a487c67e5f7\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " Nov 03 20:55:17.184056 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.183996 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-workload-socket" pod "af827a46-c0d1-4161-836f-7a487c67e5f7" (UID: "af827a46-c0d1-4161-836f-7a487c67e5f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:17.184056 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.184032 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af827a46-c0d1-4161-836f-7a487c67e5f7-kube-api-access-8klcj\" (UniqueName: \"kubernetes.io/projected/af827a46-c0d1-4161-836f-7a487c67e5f7-kube-api-access-8klcj\") pod \"af827a46-c0d1-4161-836f-7a487c67e5f7\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " Nov 03 20:55:17.184314 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.184106 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-workload-certs" pod "af827a46-c0d1-4161-836f-7a487c67e5f7" (UID: "af827a46-c0d1-4161-836f-7a487c67e5f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:17.184314 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.184119 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-data" pod "af827a46-c0d1-4161-836f-7a487c67e5f7" (UID: "af827a46-c0d1-4161-836f-7a487c67e5f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:17.184314 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.184203 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-credential-socket\") pod \"af827a46-c0d1-4161-836f-7a487c67e5f7\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " Nov 03 20:55:17.184314 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.184242 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af827a46-c0d1-4161-836f-7a487c67e5f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af827a46-c0d1-4161-836f-7a487c67e5f7-istiod-ca-cert\") pod \"af827a46-c0d1-4161-836f-7a487c67e5f7\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " Nov 03 20:55:17.184314 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.184269 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-token\" (UniqueName: \"kubernetes.io/projected/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-token\") pod \"af827a46-c0d1-4161-836f-7a487c67e5f7\" (UID: \"af827a46-c0d1-4161-836f-7a487c67e5f7\") " Nov 03 20:55:17.184581 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.184472 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-credential-socket" pod "af827a46-c0d1-4161-836f-7a487c67e5f7" (UID: "af827a46-c0d1-4161-836f-7a487c67e5f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:17.184651 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.184618 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af827a46-c0d1-4161-836f-7a487c67e5f7-istiod-ca-cert" pod "af827a46-c0d1-4161-836f-7a487c67e5f7" (UID: "af827a46-c0d1-4161-836f-7a487c67e5f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:17.184762 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.184745 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:17.184823 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.184768 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:17.184823 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.184781 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af827a46-c0d1-4161-836f-7a487c67e5f7-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:17.184823 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.184790 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:17.184823 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.184801 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:17.186017 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.185995 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-podinfo" pod "af827a46-c0d1-4161-836f-7a487c67e5f7" (UID: "af827a46-c0d1-4161-836f-7a487c67e5f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:55:17.186330 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.186314 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-envoy" pod "af827a46-c0d1-4161-836f-7a487c67e5f7" (UID: "af827a46-c0d1-4161-836f-7a487c67e5f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:17.186614 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.186583 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-token" pod "af827a46-c0d1-4161-836f-7a487c67e5f7" (UID: "af827a46-c0d1-4161-836f-7a487c67e5f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:17.186700 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.186670 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af827a46-c0d1-4161-836f-7a487c67e5f7-kube-api-access-8klcj" pod "af827a46-c0d1-4161-836f-7a487c67e5f7" (UID: "af827a46-c0d1-4161-836f-7a487c67e5f7"). InnerVolumeSpecName "kube-api-access-8klcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:17.257508 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.257476 4918 generic.go:356] "Generic (PLEG): container finished" podID="af827a46-c0d1-4161-836f-7a487c67e5f7" containerID="767048c1e9c91e3aa25c547422c936b6b9fc9b898a554f2047f61ec660beae88" exitCode=0 Nov 03 20:55:17.257629 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.257523 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" Nov 03 20:55:17.257629 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.257554 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" event={"ID":"af827a46-c0d1-4161-836f-7a487c67e5f7","Type":"ContainerDied","Data":"767048c1e9c91e3aa25c547422c936b6b9fc9b898a554f2047f61ec660beae88"} Nov 03 20:55:17.257629 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.257613 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2" event={"ID":"af827a46-c0d1-4161-836f-7a487c67e5f7","Type":"ContainerDied","Data":"d8e9d99ff35499445694ed4ef296f4d64dbbc186a6397a26f4f8dc22f4da8302"} Nov 03 20:55:17.257743 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.257636 4918 scope.go:122] "RemoveContainer" containerID="767048c1e9c91e3aa25c547422c936b6b9fc9b898a554f2047f61ec660beae88" Nov 03 20:55:17.266449 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.266433 4918 scope.go:122] "RemoveContainer" containerID="91633fb5118451936a9d374f2f754ded4ddec8a4d410ccfd1b955a25aac48f29" Nov 03 20:55:17.274680 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.274657 4918 scope.go:122] "RemoveContainer" containerID="767048c1e9c91e3aa25c547422c936b6b9fc9b898a554f2047f61ec660beae88" Nov 03 20:55:17.274948 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:55:17.274929 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"767048c1e9c91e3aa25c547422c936b6b9fc9b898a554f2047f61ec660beae88\": container with ID starting with 767048c1e9c91e3aa25c547422c936b6b9fc9b898a554f2047f61ec660beae88 not found: ID does not exist" containerID="767048c1e9c91e3aa25c547422c936b6b9fc9b898a554f2047f61ec660beae88" Nov 03 20:55:17.275045 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.274962 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"767048c1e9c91e3aa25c547422c936b6b9fc9b898a554f2047f61ec660beae88"} err="failed to get container status \"767048c1e9c91e3aa25c547422c936b6b9fc9b898a554f2047f61ec660beae88\": rpc error: code = NotFound desc = could not find container \"767048c1e9c91e3aa25c547422c936b6b9fc9b898a554f2047f61ec660beae88\": container with ID starting with 767048c1e9c91e3aa25c547422c936b6b9fc9b898a554f2047f61ec660beae88 not found: ID does not exist" Nov 03 20:55:17.275045 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.274987 4918 scope.go:122] "RemoveContainer" containerID="91633fb5118451936a9d374f2f754ded4ddec8a4d410ccfd1b955a25aac48f29" Nov 03 20:55:17.275322 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:55:17.275304 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91633fb5118451936a9d374f2f754ded4ddec8a4d410ccfd1b955a25aac48f29\": container with ID starting with 91633fb5118451936a9d374f2f754ded4ddec8a4d410ccfd1b955a25aac48f29 not found: ID does not exist" containerID="91633fb5118451936a9d374f2f754ded4ddec8a4d410ccfd1b955a25aac48f29" Nov 03 20:55:17.275378 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.275331 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91633fb5118451936a9d374f2f754ded4ddec8a4d410ccfd1b955a25aac48f29"} err="failed to get container status \"91633fb5118451936a9d374f2f754ded4ddec8a4d410ccfd1b955a25aac48f29\": rpc error: code = NotFound desc = could not find container \"91633fb5118451936a9d374f2f754ded4ddec8a4d410ccfd1b955a25aac48f29\": container with ID starting with 91633fb5118451936a9d374f2f754ded4ddec8a4d410ccfd1b955a25aac48f29 not found: ID does not exist" Nov 03 20:55:17.275783 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.275764 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2"] Nov 03 20:55:17.280574 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.280552 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bdf98686-openshift-default-68fdcbfd79-xrcx2"] Nov 03 20:55:17.286249 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.286219 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:17.286249 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.286252 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:17.286340 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.286269 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af827a46-c0d1-4161-836f-7a487c67e5f7-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:17.286340 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.286280 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8klcj\" (UniqueName: \"kubernetes.io/projected/af827a46-c0d1-4161-836f-7a487c67e5f7-kube-api-access-8klcj\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:17.703106 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.703055 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf"] Nov 03 20:55:17.703511 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.703489 4918 cpu_manager.go:414] "containerMap: removing container" podUID="af827a46-c0d1-4161-836f-7a487c67e5f7" containerName="istio-proxy" Nov 03 20:55:17.703511 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.703505 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af827a46-c0d1-4161-836f-7a487c67e5f7" containerName="istio-proxy" Nov 03 20:55:17.703730 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.703524 4918 cpu_manager.go:414] "containerMap: removing container" podUID="af827a46-c0d1-4161-836f-7a487c67e5f7" containerName="istio-proxy" Nov 03 20:55:17.703730 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.703529 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af827a46-c0d1-4161-836f-7a487c67e5f7" containerName="istio-proxy" Nov 03 20:55:17.703730 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.703608 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="af827a46-c0d1-4161-836f-7a487c67e5f7" containerName="istio-proxy" Nov 03 20:55:17.703730 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.703618 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="af827a46-c0d1-4161-836f-7a487c67e5f7" containerName="istio-proxy" Nov 03 20:55:17.708136 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.708112 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" Nov 03 20:55:17.718455 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.718428 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf"] Nov 03 20:55:17.721694 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.721669 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv"] Nov 03 20:55:17.725453 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.725433 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv" Nov 03 20:55:17.738885 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.738860 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv"] Nov 03 20:55:17.739898 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.739874 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-0fe875-serving-cert\"" Nov 03 20:55:17.740499 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.740475 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-znhr8\"" Nov 03 20:55:17.740667 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.740506 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:55:17.750064 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.750040 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq"] Nov 03 20:55:17.753537 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.753521 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq" Nov 03 20:55:17.764636 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.764613 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-0fe875-dockercfg-d4jnz\"" Nov 03 20:55:17.767477 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.767454 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq"] Nov 03 20:55:17.790583 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.790551 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:55:17.790583 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.790554 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:55:17.790738 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.790597 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-0fe875\"" Nov 03 20:55:17.790895 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.790876 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea41365e-f482-4f5c-88ec-65910c3fbe52-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf\" (UID: \"ea41365e-f482-4f5c-88ec-65910c3fbe52\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" Nov 03 20:55:17.790933 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.790917 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c995r\" (UniqueName: \"kubernetes.io/projected/ea41365e-f482-4f5c-88ec-65910c3fbe52-kube-api-access-c995r\") pod \"maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf\" (UID: \"ea41365e-f482-4f5c-88ec-65910c3fbe52\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" Nov 03 20:55:17.791049 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.791031 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea41365e-f482-4f5c-88ec-65910c3fbe52-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf\" (UID: \"ea41365e-f482-4f5c-88ec-65910c3fbe52\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" Nov 03 20:55:17.791138 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.791123 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea41365e-f482-4f5c-88ec-65910c3fbe52-metrics-certs\") pod \"maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf\" (UID: \"ea41365e-f482-4f5c-88ec-65910c3fbe52\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" Nov 03 20:55:17.892631 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.892594 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea41365e-f482-4f5c-88ec-65910c3fbe52-metrics-certs\") pod \"maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf\" (UID: \"ea41365e-f482-4f5c-88ec-65910c3fbe52\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" Nov 03 20:55:17.892801 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.892648 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f6fedd-a22a-47be-9e0f-3dd3a63dbb41-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv\" (UID: \"77f6fedd-a22a-47be-9e0f-3dd3a63dbb41\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv" Nov 03 20:55:17.892801 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.892678 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl7ks\" (UniqueName: \"kubernetes.io/projected/77f6fedd-a22a-47be-9e0f-3dd3a63dbb41-kube-api-access-fl7ks\") pod \"payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv\" (UID: \"77f6fedd-a22a-47be-9e0f-3dd3a63dbb41\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv" Nov 03 20:55:17.892801 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.892745 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/926ee044-31f5-4b9e-8a17-6ff489cf9893-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq\" (UID: \"926ee044-31f5-4b9e-8a17-6ff489cf9893\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq" Nov 03 20:55:17.892801 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.892780 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea41365e-f482-4f5c-88ec-65910c3fbe52-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf\" (UID: \"ea41365e-f482-4f5c-88ec-65910c3fbe52\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" Nov 03 20:55:17.893040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.892842 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c995r\" (UniqueName: \"kubernetes.io/projected/ea41365e-f482-4f5c-88ec-65910c3fbe52-kube-api-access-c995r\") pod \"maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf\" (UID: \"ea41365e-f482-4f5c-88ec-65910c3fbe52\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" Nov 03 20:55:17.893040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.892874 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea41365e-f482-4f5c-88ec-65910c3fbe52-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf\" (UID: \"ea41365e-f482-4f5c-88ec-65910c3fbe52\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" Nov 03 20:55:17.893040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.892898 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpz2z\" (UniqueName: \"kubernetes.io/projected/926ee044-31f5-4b9e-8a17-6ff489cf9893-kube-api-access-qpz2z\") pod \"payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq\" (UID: \"926ee044-31f5-4b9e-8a17-6ff489cf9893\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq" Nov 03 20:55:17.893543 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.893515 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea41365e-f482-4f5c-88ec-65910c3fbe52-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf\" (UID: \"ea41365e-f482-4f5c-88ec-65910c3fbe52\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" Nov 03 20:55:17.895239 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.895215 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea41365e-f482-4f5c-88ec-65910c3fbe52-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf\" (UID: \"ea41365e-f482-4f5c-88ec-65910c3fbe52\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" Nov 03 20:55:17.895341 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.895287 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea41365e-f482-4f5c-88ec-65910c3fbe52-metrics-certs\") pod \"maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf\" (UID: \"ea41365e-f482-4f5c-88ec-65910c3fbe52\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" Nov 03 20:55:17.905090 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.905068 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c995r\" (UniqueName: \"kubernetes.io/projected/ea41365e-f482-4f5c-88ec-65910c3fbe52-kube-api-access-c995r\") pod \"maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf\" (UID: \"ea41365e-f482-4f5c-88ec-65910c3fbe52\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" Nov 03 20:55:17.994360 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.994270 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f6fedd-a22a-47be-9e0f-3dd3a63dbb41-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv\" (UID: \"77f6fedd-a22a-47be-9e0f-3dd3a63dbb41\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv" Nov 03 20:55:17.994360 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.994312 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl7ks\" (UniqueName: \"kubernetes.io/projected/77f6fedd-a22a-47be-9e0f-3dd3a63dbb41-kube-api-access-fl7ks\") pod \"payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv\" (UID: \"77f6fedd-a22a-47be-9e0f-3dd3a63dbb41\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv" Nov 03 20:55:17.994360 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.994337 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/926ee044-31f5-4b9e-8a17-6ff489cf9893-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq\" (UID: \"926ee044-31f5-4b9e-8a17-6ff489cf9893\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq" Nov 03 20:55:17.994697 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.994403 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpz2z\" (UniqueName: \"kubernetes.io/projected/926ee044-31f5-4b9e-8a17-6ff489cf9893-kube-api-access-qpz2z\") pod \"payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq\" (UID: \"926ee044-31f5-4b9e-8a17-6ff489cf9893\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq" Nov 03 20:55:17.995174 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.995144 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f6fedd-a22a-47be-9e0f-3dd3a63dbb41-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv\" (UID: \"77f6fedd-a22a-47be-9e0f-3dd3a63dbb41\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv" Nov 03 20:55:17.995318 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:17.995302 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/926ee044-31f5-4b9e-8a17-6ff489cf9893-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq\" (UID: \"926ee044-31f5-4b9e-8a17-6ff489cf9893\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq" Nov 03 20:55:18.006654 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:18.006623 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl7ks\" (UniqueName: \"kubernetes.io/projected/77f6fedd-a22a-47be-9e0f-3dd3a63dbb41-kube-api-access-fl7ks\") pod \"payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv\" (UID: \"77f6fedd-a22a-47be-9e0f-3dd3a63dbb41\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv" Nov 03 20:55:18.006784 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:18.006763 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpz2z\" (UniqueName: \"kubernetes.io/projected/926ee044-31f5-4b9e-8a17-6ff489cf9893-kube-api-access-qpz2z\") pod \"payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq\" (UID: \"926ee044-31f5-4b9e-8a17-6ff489cf9893\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq" Nov 03 20:55:18.024290 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:18.024267 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" Nov 03 20:55:18.041366 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:18.041339 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv" Nov 03 20:55:18.067545 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:18.067515 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq" Nov 03 20:55:18.314891 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:18.314865 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf"] Nov 03 20:55:18.316992 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:55:18.316959 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea41365e_f482_4f5c_88ec_65910c3fbe52.slice/crio-2111a2d55cf3c08abb882260fc6e9b461daf9f32ef4a024b486f16c619e54b01 WatchSource:0}: Error finding container 2111a2d55cf3c08abb882260fc6e9b461daf9f32ef4a024b486f16c619e54b01: Status 404 returned error can't find the container with id 2111a2d55cf3c08abb882260fc6e9b461daf9f32ef4a024b486f16c619e54b01 Nov 03 20:55:18.318483 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:18.318464 4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:55:18.334588 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:18.334462 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv"] Nov 03 20:55:18.338451 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:55:18.338420 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f6fedd_a22a_47be_9e0f_3dd3a63dbb41.slice/crio-3fceb78a9d15b337bb15ba694f5932d64923a0737531bde1aafb2318f612fcd9 WatchSource:0}: Error finding container 3fceb78a9d15b337bb15ba694f5932d64923a0737531bde1aafb2318f612fcd9: Status 404 returned error can't find the container with id 3fceb78a9d15b337bb15ba694f5932d64923a0737531bde1aafb2318f612fcd9 Nov 03 20:55:18.371077 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:18.371055 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq"] Nov 03 20:55:18.372734 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:55:18.372710 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926ee044_31f5_4b9e_8a17_6ff489cf9893.slice/crio-bc901d882a32697be2c41aee44e7128c57cb6a8b5a9abd02b90a603269d6d537 WatchSource:0}: Error finding container bc901d882a32697be2c41aee44e7128c57cb6a8b5a9abd02b90a603269d6d537: Status 404 returned error can't find the container with id bc901d882a32697be2c41aee44e7128c57cb6a8b5a9abd02b90a603269d6d537 Nov 03 20:55:19.217408 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:19.217375 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af827a46-c0d1-4161-836f-7a487c67e5f7" path="/var/lib/kubelet/pods/af827a46-c0d1-4161-836f-7a487c67e5f7/volumes" Nov 03 20:55:19.270070 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:19.270028 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq" event={"ID":"926ee044-31f5-4b9e-8a17-6ff489cf9893","Type":"ContainerStarted","Data":"c33ecf93dd723c1e9472393b0d6d9971e8515cda882444374fef2fa4e16cf06a"} Nov 03 20:55:19.270261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:19.270096 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq" event={"ID":"926ee044-31f5-4b9e-8a17-6ff489cf9893","Type":"ContainerStarted","Data":"bc901d882a32697be2c41aee44e7128c57cb6a8b5a9abd02b90a603269d6d537"} Nov 03 20:55:19.270261 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:19.270201 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq" Nov 03 20:55:19.271633 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:19.271603 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv" event={"ID":"77f6fedd-a22a-47be-9e0f-3dd3a63dbb41","Type":"ContainerStarted","Data":"3f2bcd9593831e372027febe85f09d0925d2a64585355e554465871e7c752ce5"} Nov 03 20:55:19.271748 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:19.271644 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv" event={"ID":"77f6fedd-a22a-47be-9e0f-3dd3a63dbb41","Type":"ContainerStarted","Data":"3fceb78a9d15b337bb15ba694f5932d64923a0737531bde1aafb2318f612fcd9"} Nov 03 20:55:19.271748 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:19.271707 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv" Nov 03 20:55:19.273060 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:19.273038 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" event={"ID":"ea41365e-f482-4f5c-88ec-65910c3fbe52","Type":"ContainerStarted","Data":"23a696abd50c446d5486052da8e84d3e64c5facfa2e614cfacde8c1fa2503fa5"} Nov 03 20:55:19.273166 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:19.273071 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" event={"ID":"ea41365e-f482-4f5c-88ec-65910c3fbe52","Type":"ContainerStarted","Data":"2111a2d55cf3c08abb882260fc6e9b461daf9f32ef4a024b486f16c619e54b01"} Nov 03 20:55:19.273166 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:19.273129 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" Nov 03 20:55:19.289187 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:19.289145 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq" podStartSLOduration=2.289133195 podStartE2EDuration="2.289133195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:19.287901745 +0000 UTC m=+3502.219769257" watchObservedRunningTime="2026-11-03 20:55:19.289133195 +0000 UTC m=+3502.221000702" Nov 03 20:55:19.305875 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:19.305833 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv" podStartSLOduration=2.305821665 podStartE2EDuration="2.305821665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:19.304862067 +0000 UTC m=+3502.236729577" watchObservedRunningTime="2026-11-03 20:55:19.305821665 +0000 UTC m=+3502.237689201" Nov 03 20:55:19.325980 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:19.325939 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" podStartSLOduration=1.885778394 podStartE2EDuration="2.325926811s" totalImagesPullingTime="440.148417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:19.324130282 +0000 UTC m=+3502.255997796" watchObservedRunningTime="2026-11-03 20:55:19.325926811 +0000 UTC m=+3502.257794326" Nov 03 20:55:25.284673 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:25.284641 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" Nov 03 20:55:28.208340 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:55:28.208296 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:55:30.281825 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:30.281792 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq" Nov 03 20:55:30.282317 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:30.281968 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv" Nov 03 20:55:30.330975 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:30.330942 4918 generic.go:356] "Generic (PLEG): container finished" podID="5c71bc62-125e-4573-8370-cbb43ac7677e" containerID="ef365bd01bb643fed333a0774be8f0b2ccd3589bdcc1996ed64970d3151dfaac" exitCode=0 Nov 03 20:55:30.331168 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:30.331015 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" event={"ID":"5c71bc62-125e-4573-8370-cbb43ac7677e","Type":"ContainerDied","Data":"ef365bd01bb643fed333a0774be8f0b2ccd3589bdcc1996ed64970d3151dfaac"} Nov 03 20:55:30.331513 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:30.331495 4918 scope.go:122] "RemoveContainer" containerID="ef365bd01bb643fed333a0774be8f0b2ccd3589bdcc1996ed64970d3151dfaac" Nov 03 20:55:30.332849 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:30.332819 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:55:30.332948 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:30.332900 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:55:30.333014 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:30.332950 4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:55:31.251771 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:31.251733 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:31.350208 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:31.350151 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" event={"ID":"5c71bc62-125e-4573-8370-cbb43ac7677e","Type":"ContainerStarted","Data":"b615c2096325f1e799eaddefcc1cc8e86feb0828613e8558ec06449ac26df57e"} Nov 03 20:55:31.847430 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:31.847383 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:31.848986 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:31.848952 4918 patch_prober.go:29] interesting pod/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 20:55:31.849124 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:31.849030 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" podUID="5c71bc62-125e-4573-8370-cbb43ac7677e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 20:55:32.847747 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:32.847709 4918 patch_prober.go:29] interesting pod/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 20:55:32.848452 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:32.848409 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" podUID="5c71bc62-125e-4573-8370-cbb43ac7677e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 20:55:33.847892 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:33.847851 4918 patch_prober.go:29] interesting pod/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 20:55:33.848333 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:33.847909 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" podUID="5c71bc62-125e-4573-8370-cbb43ac7677e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 20:55:34.847591 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:34.847544 4918 patch_prober.go:29] interesting pod/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 20:55:34.847787 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:34.847622 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" podUID="5c71bc62-125e-4573-8370-cbb43ac7677e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 20:55:35.847955 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:35.847919 4918 patch_prober.go:29] interesting pod/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 20:55:35.848382 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:35.847982 4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" podUID="5c71bc62-125e-4573-8370-cbb43ac7677e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 20:55:36.852103 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:36.852071 4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:36.852527 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:36.852463 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:36.853223 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:55:36.853203 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:55:58.208054 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:55:58.208017 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:56:28.209106 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:56:28.209065 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:56:57.706251 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:56:57.706221 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:56:57.709967 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:56:57.709940 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 20:56:58.207956 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:56:58.207916 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:57:01.779783 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:01.779742 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv"] Nov 03 20:57:01.791946 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:01.791907 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq"] Nov 03 20:57:06.780352 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:06.780278 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv" podUID="77f6fedd-a22a-47be-9e0f-3dd3a63dbb41" containerName="payload-processing" containerID="cri-o://3f2bcd9593831e372027febe85f09d0925d2a64585355e554465871e7c752ce5" gracePeriod=25 Nov 03 20:57:06.793039 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:06.792968 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq" podUID="926ee044-31f5-4b9e-8a17-6ff489cf9893" containerName="payload-pre-processing" containerID="cri-o://c33ecf93dd723c1e9472393b0d6d9971e8515cda882444374fef2fa4e16cf06a" gracePeriod=25 Nov 03 20:57:07.056410 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.056377 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv" Nov 03 20:57:07.075012 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.074983 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq" Nov 03 20:57:07.191403 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.191370 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/926ee044-31f5-4b9e-8a17-6ff489cf9893-kube-api-access-qpz2z\" (UniqueName: \"kubernetes.io/projected/926ee044-31f5-4b9e-8a17-6ff489cf9893-kube-api-access-qpz2z\") pod \"926ee044-31f5-4b9e-8a17-6ff489cf9893\" (UID: \"926ee044-31f5-4b9e-8a17-6ff489cf9893\") " Nov 03 20:57:07.191615 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.191417 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/926ee044-31f5-4b9e-8a17-6ff489cf9893-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/926ee044-31f5-4b9e-8a17-6ff489cf9893-plugins-config-volume\") pod \"926ee044-31f5-4b9e-8a17-6ff489cf9893\" (UID: \"926ee044-31f5-4b9e-8a17-6ff489cf9893\") " Nov 03 20:57:07.191615 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.191522 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f6fedd-a22a-47be-9e0f-3dd3a63dbb41-kube-api-access-fl7ks\" (UniqueName: \"kubernetes.io/projected/77f6fedd-a22a-47be-9e0f-3dd3a63dbb41-kube-api-access-fl7ks\") pod \"77f6fedd-a22a-47be-9e0f-3dd3a63dbb41\" (UID: \"77f6fedd-a22a-47be-9e0f-3dd3a63dbb41\") " Nov 03 20:57:07.191615 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.191559 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f6fedd-a22a-47be-9e0f-3dd3a63dbb41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f6fedd-a22a-47be-9e0f-3dd3a63dbb41-plugins-config-volume\") pod \"77f6fedd-a22a-47be-9e0f-3dd3a63dbb41\" (UID: \"77f6fedd-a22a-47be-9e0f-3dd3a63dbb41\") " Nov 03 20:57:07.191922 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.191889 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926ee044-31f5-4b9e-8a17-6ff489cf9893-plugins-config-volume" pod "926ee044-31f5-4b9e-8a17-6ff489cf9893" (UID: "926ee044-31f5-4b9e-8a17-6ff489cf9893"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:07.192052 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.191891 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f6fedd-a22a-47be-9e0f-3dd3a63dbb41-plugins-config-volume" pod "77f6fedd-a22a-47be-9e0f-3dd3a63dbb41" (UID: "77f6fedd-a22a-47be-9e0f-3dd3a63dbb41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:07.193581 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.193559 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926ee044-31f5-4b9e-8a17-6ff489cf9893-kube-api-access-qpz2z" pod "926ee044-31f5-4b9e-8a17-6ff489cf9893" (UID: "926ee044-31f5-4b9e-8a17-6ff489cf9893"). InnerVolumeSpecName "kube-api-access-qpz2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:07.193651 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.193617 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f6fedd-a22a-47be-9e0f-3dd3a63dbb41-kube-api-access-fl7ks" pod "77f6fedd-a22a-47be-9e0f-3dd3a63dbb41" (UID: "77f6fedd-a22a-47be-9e0f-3dd3a63dbb41"). InnerVolumeSpecName "kube-api-access-fl7ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:07.292972 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.292895 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl7ks\" (UniqueName: \"kubernetes.io/projected/77f6fedd-a22a-47be-9e0f-3dd3a63dbb41-kube-api-access-fl7ks\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:07.292972 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.292930 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f6fedd-a22a-47be-9e0f-3dd3a63dbb41-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:07.292972 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.292947 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpz2z\" (UniqueName: \"kubernetes.io/projected/926ee044-31f5-4b9e-8a17-6ff489cf9893-kube-api-access-qpz2z\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:07.292972 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.292960 4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/926ee044-31f5-4b9e-8a17-6ff489cf9893-plugins-config-volume\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:07.709323 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.709289 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf"] Nov 03 20:57:07.709548 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.709503 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" podUID="ea41365e-f482-4f5c-88ec-65910c3fbe52" containerName="maas-api" containerID="cri-o://23a696abd50c446d5486052da8e84d3e64c5facfa2e614cfacde8c1fa2503fa5" gracePeriod=30 Nov 03 20:57:07.814155 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.814125 4918 generic.go:356] "Generic (PLEG): container finished" podID="926ee044-31f5-4b9e-8a17-6ff489cf9893" containerID="c33ecf93dd723c1e9472393b0d6d9971e8515cda882444374fef2fa4e16cf06a" exitCode=0 Nov 03 20:57:07.814591 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.814164 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq" Nov 03 20:57:07.814591 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.814207 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq" event={"ID":"926ee044-31f5-4b9e-8a17-6ff489cf9893","Type":"ContainerDied","Data":"c33ecf93dd723c1e9472393b0d6d9971e8515cda882444374fef2fa4e16cf06a"} Nov 03 20:57:07.814591 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.814264 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq" event={"ID":"926ee044-31f5-4b9e-8a17-6ff489cf9893","Type":"ContainerDied","Data":"bc901d882a32697be2c41aee44e7128c57cb6a8b5a9abd02b90a603269d6d537"} Nov 03 20:57:07.814591 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.814291 4918 scope.go:122] "RemoveContainer" containerID="c33ecf93dd723c1e9472393b0d6d9971e8515cda882444374fef2fa4e16cf06a" Nov 03 20:57:07.816102 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.815935 4918 generic.go:356] "Generic (PLEG): container finished" podID="77f6fedd-a22a-47be-9e0f-3dd3a63dbb41" containerID="3f2bcd9593831e372027febe85f09d0925d2a64585355e554465871e7c752ce5" exitCode=0 Nov 03 20:57:07.816102 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.815987 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv" event={"ID":"77f6fedd-a22a-47be-9e0f-3dd3a63dbb41","Type":"ContainerDied","Data":"3f2bcd9593831e372027febe85f09d0925d2a64585355e554465871e7c752ce5"} Nov 03 20:57:07.816102 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.816006 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv" Nov 03 20:57:07.816102 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.816044 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv" event={"ID":"77f6fedd-a22a-47be-9e0f-3dd3a63dbb41","Type":"ContainerDied","Data":"3fceb78a9d15b337bb15ba694f5932d64923a0737531bde1aafb2318f612fcd9"} Nov 03 20:57:07.823445 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.823428 4918 scope.go:122] "RemoveContainer" containerID="c33ecf93dd723c1e9472393b0d6d9971e8515cda882444374fef2fa4e16cf06a" Nov 03 20:57:07.823716 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:57:07.823695 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33ecf93dd723c1e9472393b0d6d9971e8515cda882444374fef2fa4e16cf06a\": container with ID starting with c33ecf93dd723c1e9472393b0d6d9971e8515cda882444374fef2fa4e16cf06a not found: ID does not exist" containerID="c33ecf93dd723c1e9472393b0d6d9971e8515cda882444374fef2fa4e16cf06a" Nov 03 20:57:07.823800 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.823728 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33ecf93dd723c1e9472393b0d6d9971e8515cda882444374fef2fa4e16cf06a"} err="failed to get container status \"c33ecf93dd723c1e9472393b0d6d9971e8515cda882444374fef2fa4e16cf06a\": rpc error: code = NotFound desc = could not find container \"c33ecf93dd723c1e9472393b0d6d9971e8515cda882444374fef2fa4e16cf06a\": container with ID starting with c33ecf93dd723c1e9472393b0d6d9971e8515cda882444374fef2fa4e16cf06a not found: ID does not exist" Nov 03 20:57:07.823800 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.823751 4918 scope.go:122] "RemoveContainer" containerID="3f2bcd9593831e372027febe85f09d0925d2a64585355e554465871e7c752ce5" Nov 03 20:57:07.832678 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.832657 4918 scope.go:122] "RemoveContainer" containerID="3f2bcd9593831e372027febe85f09d0925d2a64585355e554465871e7c752ce5" Nov 03 20:57:07.833043 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:57:07.833005 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2bcd9593831e372027febe85f09d0925d2a64585355e554465871e7c752ce5\": container with ID starting with 3f2bcd9593831e372027febe85f09d0925d2a64585355e554465871e7c752ce5 not found: ID does not exist" containerID="3f2bcd9593831e372027febe85f09d0925d2a64585355e554465871e7c752ce5" Nov 03 20:57:07.833043 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.833047 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2bcd9593831e372027febe85f09d0925d2a64585355e554465871e7c752ce5"} err="failed to get container status \"3f2bcd9593831e372027febe85f09d0925d2a64585355e554465871e7c752ce5\": rpc error: code = NotFound desc = could not find container \"3f2bcd9593831e372027febe85f09d0925d2a64585355e554465871e7c752ce5\": container with ID starting with 3f2bcd9593831e372027febe85f09d0925d2a64585355e554465871e7c752ce5 not found: ID does not exist" Nov 03 20:57:07.835260 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.835238 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq"] Nov 03 20:57:07.842712 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.842688 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0fe875-776447f559-8zspq"] Nov 03 20:57:07.856943 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.856916 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv"] Nov 03 20:57:07.861047 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.861021 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0fe875-846b69cfcd-r5wjv"] Nov 03 20:57:07.970829 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:07.970762 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" Nov 03 20:57:08.102028 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.101991 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea41365e-f482-4f5c-88ec-65910c3fbe52-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea41365e-f482-4f5c-88ec-65910c3fbe52-metrics-certs\") pod \"ea41365e-f482-4f5c-88ec-65910c3fbe52\" (UID: \"ea41365e-f482-4f5c-88ec-65910c3fbe52\") " Nov 03 20:57:08.102225 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.102089 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea41365e-f482-4f5c-88ec-65910c3fbe52-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea41365e-f482-4f5c-88ec-65910c3fbe52-openshift-service-ca\") pod \"ea41365e-f482-4f5c-88ec-65910c3fbe52\" (UID: \"ea41365e-f482-4f5c-88ec-65910c3fbe52\") " Nov 03 20:57:08.102225 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.102123 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea41365e-f482-4f5c-88ec-65910c3fbe52-kube-api-access-c995r\" (UniqueName: \"kubernetes.io/projected/ea41365e-f482-4f5c-88ec-65910c3fbe52-kube-api-access-c995r\") pod \"ea41365e-f482-4f5c-88ec-65910c3fbe52\" (UID: \"ea41365e-f482-4f5c-88ec-65910c3fbe52\") " Nov 03 20:57:08.102225 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.102147 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea41365e-f482-4f5c-88ec-65910c3fbe52-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea41365e-f482-4f5c-88ec-65910c3fbe52-maas-api-tls\") pod \"ea41365e-f482-4f5c-88ec-65910c3fbe52\" (UID: \"ea41365e-f482-4f5c-88ec-65910c3fbe52\") " Nov 03 20:57:08.102608 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.102575 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea41365e-f482-4f5c-88ec-65910c3fbe52-openshift-service-ca" pod "ea41365e-f482-4f5c-88ec-65910c3fbe52" (UID: "ea41365e-f482-4f5c-88ec-65910c3fbe52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:08.104288 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.104253 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea41365e-f482-4f5c-88ec-65910c3fbe52-metrics-certs" pod "ea41365e-f482-4f5c-88ec-65910c3fbe52" (UID: "ea41365e-f482-4f5c-88ec-65910c3fbe52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:57:08.104406 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.104308 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea41365e-f482-4f5c-88ec-65910c3fbe52-maas-api-tls" pod "ea41365e-f482-4f5c-88ec-65910c3fbe52" (UID: "ea41365e-f482-4f5c-88ec-65910c3fbe52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:57:08.104406 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.104363 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea41365e-f482-4f5c-88ec-65910c3fbe52-kube-api-access-c995r" pod "ea41365e-f482-4f5c-88ec-65910c3fbe52" (UID: "ea41365e-f482-4f5c-88ec-65910c3fbe52"). InnerVolumeSpecName "kube-api-access-c995r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:08.203768 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.203730 4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea41365e-f482-4f5c-88ec-65910c3fbe52-metrics-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:08.203768 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.203767 4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea41365e-f482-4f5c-88ec-65910c3fbe52-openshift-service-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:08.203976 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.203783 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c995r\" (UniqueName: \"kubernetes.io/projected/ea41365e-f482-4f5c-88ec-65910c3fbe52-kube-api-access-c995r\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:08.203976 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.203796 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea41365e-f482-4f5c-88ec-65910c3fbe52-maas-api-tls\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:08.823677 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.823640 4918 generic.go:356] "Generic (PLEG): container finished" podID="ea41365e-f482-4f5c-88ec-65910c3fbe52" containerID="23a696abd50c446d5486052da8e84d3e64c5facfa2e614cfacde8c1fa2503fa5" exitCode=0 Nov 03 20:57:08.824167 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.823705 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" Nov 03 20:57:08.824167 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.823727 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" event={"ID":"ea41365e-f482-4f5c-88ec-65910c3fbe52","Type":"ContainerDied","Data":"23a696abd50c446d5486052da8e84d3e64c5facfa2e614cfacde8c1fa2503fa5"} Nov 03 20:57:08.824167 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.823782 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf" event={"ID":"ea41365e-f482-4f5c-88ec-65910c3fbe52","Type":"ContainerDied","Data":"2111a2d55cf3c08abb882260fc6e9b461daf9f32ef4a024b486f16c619e54b01"} Nov 03 20:57:08.824167 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.823807 4918 scope.go:122] "RemoveContainer" containerID="23a696abd50c446d5486052da8e84d3e64c5facfa2e614cfacde8c1fa2503fa5" Nov 03 20:57:08.835087 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.835068 4918 scope.go:122] "RemoveContainer" containerID="23a696abd50c446d5486052da8e84d3e64c5facfa2e614cfacde8c1fa2503fa5" Nov 03 20:57:08.835411 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:57:08.835390 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23a696abd50c446d5486052da8e84d3e64c5facfa2e614cfacde8c1fa2503fa5\": container with ID starting with 23a696abd50c446d5486052da8e84d3e64c5facfa2e614cfacde8c1fa2503fa5 not found: ID does not exist" containerID="23a696abd50c446d5486052da8e84d3e64c5facfa2e614cfacde8c1fa2503fa5" Nov 03 20:57:08.835487 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.835421 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23a696abd50c446d5486052da8e84d3e64c5facfa2e614cfacde8c1fa2503fa5"} err="failed to get container status \"23a696abd50c446d5486052da8e84d3e64c5facfa2e614cfacde8c1fa2503fa5\": rpc error: code = NotFound desc = could not find container \"23a696abd50c446d5486052da8e84d3e64c5facfa2e614cfacde8c1fa2503fa5\": container with ID starting with 23a696abd50c446d5486052da8e84d3e64c5facfa2e614cfacde8c1fa2503fa5 not found: ID does not exist" Nov 03 20:57:08.848036 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.848003 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf"] Nov 03 20:57:08.851773 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:08.851751 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-0fe875-7cf6f98859-qzcjf"] Nov 03 20:57:09.213412 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:09.213335 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f6fedd-a22a-47be-9e0f-3dd3a63dbb41" path="/var/lib/kubelet/pods/77f6fedd-a22a-47be-9e0f-3dd3a63dbb41/volumes" Nov 03 20:57:09.213739 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:09.213725 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="926ee044-31f5-4b9e-8a17-6ff489cf9893" path="/var/lib/kubelet/pods/926ee044-31f5-4b9e-8a17-6ff489cf9893/volumes" Nov 03 20:57:09.214100 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:09.214088 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea41365e-f482-4f5c-88ec-65910c3fbe52" path="/var/lib/kubelet/pods/ea41365e-f482-4f5c-88ec-65910c3fbe52/volumes" Nov 03 20:57:15.146478 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:15.146419 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng"] Nov 03 20:57:15.147024 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:15.146701 4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" podUID="5c71bc62-125e-4573-8370-cbb43ac7677e" containerName="istio-proxy" containerID="cri-o://b615c2096325f1e799eaddefcc1cc8e86feb0828613e8558ec06449ac26df57e" gracePeriod=30 Nov 03 20:57:19.933345 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:19.933300 4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8bvn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:57:19.934041 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:19.934005 4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8bvn4" podUID="ea819fe0-461b-4952-a40e-7a47e8f51580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:57:20.399816 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.399789 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:57:20.422657 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.422632 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c71bc62-125e-4573-8370-cbb43ac7677e-kube-api-access-69rsp\" (UniqueName: \"kubernetes.io/projected/5c71bc62-125e-4573-8370-cbb43ac7677e-kube-api-access-69rsp\") pod \"5c71bc62-125e-4573-8370-cbb43ac7677e\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " Nov 03 20:57:20.422772 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.422699 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-credential-socket\") pod \"5c71bc62-125e-4573-8370-cbb43ac7677e\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " Nov 03 20:57:20.422772 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.422748 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-podinfo\") pod \"5c71bc62-125e-4573-8370-cbb43ac7677e\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " Nov 03 20:57:20.422865 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.422789 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-token\" (UniqueName: \"kubernetes.io/projected/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-token\") pod \"5c71bc62-125e-4573-8370-cbb43ac7677e\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " Nov 03 20:57:20.422865 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.422836 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-workload-certs\") pod \"5c71bc62-125e-4573-8370-cbb43ac7677e\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " Nov 03 20:57:20.422973 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.422868 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-data\") pod \"5c71bc62-125e-4573-8370-cbb43ac7677e\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " Nov 03 20:57:20.422973 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.422909 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-envoy\") pod \"5c71bc62-125e-4573-8370-cbb43ac7677e\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " Nov 03 20:57:20.423083 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.422971 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-workload-socket\") pod \"5c71bc62-125e-4573-8370-cbb43ac7677e\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " Nov 03 20:57:20.423083 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.423004 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c71bc62-125e-4573-8370-cbb43ac7677e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c71bc62-125e-4573-8370-cbb43ac7677e-istiod-ca-cert\") pod \"5c71bc62-125e-4573-8370-cbb43ac7677e\" (UID: \"5c71bc62-125e-4573-8370-cbb43ac7677e\") " Nov 03 20:57:20.423083 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.423052 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-credential-socket" pod "5c71bc62-125e-4573-8370-cbb43ac7677e" (UID: "5c71bc62-125e-4573-8370-cbb43ac7677e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:20.423332 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.423137 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-workload-certs" pod "5c71bc62-125e-4573-8370-cbb43ac7677e" (UID: "5c71bc62-125e-4573-8370-cbb43ac7677e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:20.423573 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.423401 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-workload-socket" pod "5c71bc62-125e-4573-8370-cbb43ac7677e" (UID: "5c71bc62-125e-4573-8370-cbb43ac7677e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:20.424009 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.423420 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-data" pod "5c71bc62-125e-4573-8370-cbb43ac7677e" (UID: "5c71bc62-125e-4573-8370-cbb43ac7677e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:20.424009 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.423753 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c71bc62-125e-4573-8370-cbb43ac7677e-istiod-ca-cert" pod "5c71bc62-125e-4573-8370-cbb43ac7677e" (UID: "5c71bc62-125e-4573-8370-cbb43ac7677e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:20.424009 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.423519 4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-credential-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:20.424009 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.423847 4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-workload-certs\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:20.425816 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.425772 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-token" pod "5c71bc62-125e-4573-8370-cbb43ac7677e" (UID: "5c71bc62-125e-4573-8370-cbb43ac7677e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:20.425929 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.425897 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-podinfo" pod "5c71bc62-125e-4573-8370-cbb43ac7677e" (UID: "5c71bc62-125e-4573-8370-cbb43ac7677e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:57:20.426033 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.426013 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c71bc62-125e-4573-8370-cbb43ac7677e-kube-api-access-69rsp" pod "5c71bc62-125e-4573-8370-cbb43ac7677e" (UID: "5c71bc62-125e-4573-8370-cbb43ac7677e"). InnerVolumeSpecName "kube-api-access-69rsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:20.427129 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.427099 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-envoy" pod "5c71bc62-125e-4573-8370-cbb43ac7677e" (UID: "5c71bc62-125e-4573-8370-cbb43ac7677e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:20.525040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.524964 4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-podinfo\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:20.525040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.525001 4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-token\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:20.525040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.525011 4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-data\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:20.525040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.525020 4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-istio-envoy\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:20.525040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.525029 4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c71bc62-125e-4573-8370-cbb43ac7677e-workload-socket\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:20.525040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.525039 4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c71bc62-125e-4573-8370-cbb43ac7677e-istiod-ca-cert\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:20.525040 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.525048 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69rsp\" (UniqueName: \"kubernetes.io/projected/5c71bc62-125e-4573-8370-cbb43ac7677e-kube-api-access-69rsp\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:20.892927 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.892891 4918 generic.go:356] "Generic (PLEG): container finished" podID="5c71bc62-125e-4573-8370-cbb43ac7677e" containerID="b615c2096325f1e799eaddefcc1cc8e86feb0828613e8558ec06449ac26df57e" exitCode=0 Nov 03 20:57:20.893110 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.892945 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" Nov 03 20:57:20.893110 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.892972 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" event={"ID":"5c71bc62-125e-4573-8370-cbb43ac7677e","Type":"ContainerDied","Data":"b615c2096325f1e799eaddefcc1cc8e86feb0828613e8558ec06449ac26df57e"} Nov 03 20:57:20.893110 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.893028 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng" event={"ID":"5c71bc62-125e-4573-8370-cbb43ac7677e","Type":"ContainerDied","Data":"eb8616494079227f2c4dec2a5687e291f8e3a2de2cb1d8e70d6d8a5f0e478e7f"} Nov 03 20:57:20.893110 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.893050 4918 scope.go:122] "RemoveContainer" containerID="b615c2096325f1e799eaddefcc1cc8e86feb0828613e8558ec06449ac26df57e" Nov 03 20:57:20.903136 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.903115 4918 scope.go:122] "RemoveContainer" containerID="ef365bd01bb643fed333a0774be8f0b2ccd3589bdcc1996ed64970d3151dfaac" Nov 03 20:57:20.911264 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.911242 4918 scope.go:122] "RemoveContainer" containerID="b615c2096325f1e799eaddefcc1cc8e86feb0828613e8558ec06449ac26df57e" Nov 03 20:57:20.911541 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:57:20.911522 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b615c2096325f1e799eaddefcc1cc8e86feb0828613e8558ec06449ac26df57e\": container with ID starting with b615c2096325f1e799eaddefcc1cc8e86feb0828613e8558ec06449ac26df57e not found: ID does not exist" containerID="b615c2096325f1e799eaddefcc1cc8e86feb0828613e8558ec06449ac26df57e" Nov 03 20:57:20.911588 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.911551 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b615c2096325f1e799eaddefcc1cc8e86feb0828613e8558ec06449ac26df57e"} err="failed to get container status \"b615c2096325f1e799eaddefcc1cc8e86feb0828613e8558ec06449ac26df57e\": rpc error: code = NotFound desc = could not find container \"b615c2096325f1e799eaddefcc1cc8e86feb0828613e8558ec06449ac26df57e\": container with ID starting with b615c2096325f1e799eaddefcc1cc8e86feb0828613e8558ec06449ac26df57e not found: ID does not exist" Nov 03 20:57:20.911588 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.911569 4918 scope.go:122] "RemoveContainer" containerID="ef365bd01bb643fed333a0774be8f0b2ccd3589bdcc1996ed64970d3151dfaac" Nov 03 20:57:20.911800 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:57:20.911786 4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef365bd01bb643fed333a0774be8f0b2ccd3589bdcc1996ed64970d3151dfaac\": container with ID starting with ef365bd01bb643fed333a0774be8f0b2ccd3589bdcc1996ed64970d3151dfaac not found: ID does not exist" containerID="ef365bd01bb643fed333a0774be8f0b2ccd3589bdcc1996ed64970d3151dfaac" Nov 03 20:57:20.911846 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.911804 4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef365bd01bb643fed333a0774be8f0b2ccd3589bdcc1996ed64970d3151dfaac"} err="failed to get container status \"ef365bd01bb643fed333a0774be8f0b2ccd3589bdcc1996ed64970d3151dfaac\": rpc error: code = NotFound desc = could not find container \"ef365bd01bb643fed333a0774be8f0b2ccd3589bdcc1996ed64970d3151dfaac\": container with ID starting with ef365bd01bb643fed333a0774be8f0b2ccd3589bdcc1996ed64970d3151dfaac not found: ID does not exist" Nov 03 20:57:20.915622 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.915599 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng"] Nov 03 20:57:20.920164 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:20.920143 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0fe875-openshift-default-5578dd6fbf-h9xng"] Nov 03 20:57:21.213102 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:21.213030 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c71bc62-125e-4573-8370-cbb43ac7677e" path="/var/lib/kubelet/pods/5c71bc62-125e-4573-8370-cbb43ac7677e/volumes" Nov 03 20:57:26.987713 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.987671 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vnd4z"] Nov 03 20:57:26.988270 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.988250 4918 cpu_manager.go:414] "containerMap: removing container" podUID="ea41365e-f482-4f5c-88ec-65910c3fbe52" containerName="maas-api" Nov 03 20:57:26.988353 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.988271 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea41365e-f482-4f5c-88ec-65910c3fbe52" containerName="maas-api" Nov 03 20:57:26.988353 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.988287 4918 cpu_manager.go:414] "containerMap: removing container" podUID="5c71bc62-125e-4573-8370-cbb43ac7677e" containerName="istio-proxy" Nov 03 20:57:26.988353 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.988295 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c71bc62-125e-4573-8370-cbb43ac7677e" containerName="istio-proxy" Nov 03 20:57:26.988353 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.988334 4918 cpu_manager.go:414] "containerMap: removing container" podUID="926ee044-31f5-4b9e-8a17-6ff489cf9893" containerName="payload-pre-processing" Nov 03 20:57:26.988353 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.988341 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="926ee044-31f5-4b9e-8a17-6ff489cf9893" containerName="payload-pre-processing" Nov 03 20:57:26.988637 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.988362 4918 cpu_manager.go:414] "containerMap: removing container" podUID="5c71bc62-125e-4573-8370-cbb43ac7677e" containerName="istio-proxy" Nov 03 20:57:26.988637 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.988370 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c71bc62-125e-4573-8370-cbb43ac7677e" containerName="istio-proxy" Nov 03 20:57:26.988637 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.988391 4918 cpu_manager.go:414] "containerMap: removing container" podUID="77f6fedd-a22a-47be-9e0f-3dd3a63dbb41" containerName="payload-processing" Nov 03 20:57:26.988637 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.988400 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f6fedd-a22a-47be-9e0f-3dd3a63dbb41" containerName="payload-processing" Nov 03 20:57:26.988637 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.988499 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f6fedd-a22a-47be-9e0f-3dd3a63dbb41" containerName="payload-processing" Nov 03 20:57:26.988637 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.988515 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea41365e-f482-4f5c-88ec-65910c3fbe52" containerName="maas-api" Nov 03 20:57:26.988637 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.988528 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c71bc62-125e-4573-8370-cbb43ac7677e" containerName="istio-proxy" Nov 03 20:57:26.988637 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.988540 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="926ee044-31f5-4b9e-8a17-6ff489cf9893" containerName="payload-pre-processing" Nov 03 20:57:26.988637 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.988551 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c71bc62-125e-4573-8370-cbb43ac7677e" containerName="istio-proxy" Nov 03 20:57:26.993349 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.993328 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vnd4z" Nov 03 20:57:27.002216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.999032 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:57:27.002216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.999081 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:57:27.002216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.999089 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:57:27.002216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:26.999156 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xhgp6\"" Nov 03 20:57:27.002216 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:27.000627 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vnd4z"] Nov 03 20:57:27.030726 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:27.030698 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e8de4ed-6171-40e3-93d7-4f057f5209c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-vnd4z\" (UID: \"3e8de4ed-6171-40e3-93d7-4f057f5209c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vnd4z" Nov 03 20:57:27.030847 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:27.030748 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e8de4ed-6171-40e3-93d7-4f057f5209c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-vnd4z\" (UID: \"3e8de4ed-6171-40e3-93d7-4f057f5209c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vnd4z" Nov 03 20:57:27.030847 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:27.030790 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8w4v\" (UniqueName: \"kubernetes.io/projected/3e8de4ed-6171-40e3-93d7-4f057f5209c2-kube-api-access-k8w4v\") pod \"maas-api-revoke-sub-f03629db-vnd4z\" (UID: \"3e8de4ed-6171-40e3-93d7-4f057f5209c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vnd4z" Nov 03 20:57:27.131764 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:27.131730 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8w4v\" (UniqueName: \"kubernetes.io/projected/3e8de4ed-6171-40e3-93d7-4f057f5209c2-kube-api-access-k8w4v\") pod \"maas-api-revoke-sub-f03629db-vnd4z\" (UID: \"3e8de4ed-6171-40e3-93d7-4f057f5209c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vnd4z" Nov 03 20:57:27.131967 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:27.131821 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e8de4ed-6171-40e3-93d7-4f057f5209c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-vnd4z\" (UID: \"3e8de4ed-6171-40e3-93d7-4f057f5209c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vnd4z" Nov 03 20:57:27.131967 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:27.131850 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e8de4ed-6171-40e3-93d7-4f057f5209c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-vnd4z\" (UID: \"3e8de4ed-6171-40e3-93d7-4f057f5209c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vnd4z" Nov 03 20:57:27.132555 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:27.132530 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e8de4ed-6171-40e3-93d7-4f057f5209c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-vnd4z\" (UID: \"3e8de4ed-6171-40e3-93d7-4f057f5209c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vnd4z" Nov 03 20:57:27.134239 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:27.134218 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e8de4ed-6171-40e3-93d7-4f057f5209c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-vnd4z\" (UID: \"3e8de4ed-6171-40e3-93d7-4f057f5209c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vnd4z" Nov 03 20:57:27.144266 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:27.144246 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8w4v\" (UniqueName: \"kubernetes.io/projected/3e8de4ed-6171-40e3-93d7-4f057f5209c2-kube-api-access-k8w4v\") pod \"maas-api-revoke-sub-f03629db-vnd4z\" (UID: \"3e8de4ed-6171-40e3-93d7-4f057f5209c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vnd4z" Nov 03 20:57:27.307533 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:27.307450 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vnd4z" Nov 03 20:57:27.584972 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:27.584944 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vnd4z"] Nov 03 20:57:27.587504 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:57:27.587473 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8de4ed_6171_40e3_93d7_4f057f5209c2.slice/crio-0b389fee9a97f719737032ac2ec9ef8c1c2aed3f78182c4acb996d669cc8f991 WatchSource:0}: Error finding container 0b389fee9a97f719737032ac2ec9ef8c1c2aed3f78182c4acb996d669cc8f991: Status 404 returned error can't find the container with id 0b389fee9a97f719737032ac2ec9ef8c1c2aed3f78182c4acb996d669cc8f991 Nov 03 20:57:27.929947 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:27.929915 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vnd4z" event={"ID":"3e8de4ed-6171-40e3-93d7-4f057f5209c2","Type":"ContainerStarted","Data":"0b389fee9a97f719737032ac2ec9ef8c1c2aed3f78182c4acb996d669cc8f991"} Nov 03 20:57:28.207704 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:57:28.207625 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:57:28.936041 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:28.936001 4918 generic.go:356] "Generic (PLEG): container finished" podID="3e8de4ed-6171-40e3-93d7-4f057f5209c2" containerID="ca436d6767d82aa69eccb28df2f793a09f2bdada27c2b3ddc9e83fa7cc2a2802" exitCode=0 Nov 03 20:57:28.936332 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:28.936069 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vnd4z" event={"ID":"3e8de4ed-6171-40e3-93d7-4f057f5209c2","Type":"ContainerDied","Data":"ca436d6767d82aa69eccb28df2f793a09f2bdada27c2b3ddc9e83fa7cc2a2802"} Nov 03 20:57:30.081477 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:30.081450 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vnd4z" Nov 03 20:57:30.160712 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:30.160678 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8de4ed-6171-40e3-93d7-4f057f5209c2-kube-api-access-k8w4v\" (UniqueName: \"kubernetes.io/projected/3e8de4ed-6171-40e3-93d7-4f057f5209c2-kube-api-access-k8w4v\") pod \"3e8de4ed-6171-40e3-93d7-4f057f5209c2\" (UID: \"3e8de4ed-6171-40e3-93d7-4f057f5209c2\") " Nov 03 20:57:30.160998 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:30.160723 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e8de4ed-6171-40e3-93d7-4f057f5209c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e8de4ed-6171-40e3-93d7-4f057f5209c2-maas-api-xks-ca\") pod \"3e8de4ed-6171-40e3-93d7-4f057f5209c2\" (UID: \"3e8de4ed-6171-40e3-93d7-4f057f5209c2\") " Nov 03 20:57:30.160998 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:30.160774 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e8de4ed-6171-40e3-93d7-4f057f5209c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e8de4ed-6171-40e3-93d7-4f057f5209c2-maas-api-openshift-ca\") pod \"3e8de4ed-6171-40e3-93d7-4f057f5209c2\" (UID: \"3e8de4ed-6171-40e3-93d7-4f057f5209c2\") " Nov 03 20:57:30.161330 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:30.161297 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e8de4ed-6171-40e3-93d7-4f057f5209c2-maas-api-openshift-ca" pod "3e8de4ed-6171-40e3-93d7-4f057f5209c2" (UID: "3e8de4ed-6171-40e3-93d7-4f057f5209c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:30.162843 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:30.162811 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e8de4ed-6171-40e3-93d7-4f057f5209c2-maas-api-xks-ca" pod "3e8de4ed-6171-40e3-93d7-4f057f5209c2" (UID: "3e8de4ed-6171-40e3-93d7-4f057f5209c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:57:30.162943 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:30.162904 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8de4ed-6171-40e3-93d7-4f057f5209c2-kube-api-access-k8w4v" pod "3e8de4ed-6171-40e3-93d7-4f057f5209c2" (UID: "3e8de4ed-6171-40e3-93d7-4f057f5209c2"). InnerVolumeSpecName "kube-api-access-k8w4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:30.261578 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:30.261503 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8w4v\" (UniqueName: \"kubernetes.io/projected/3e8de4ed-6171-40e3-93d7-4f057f5209c2-kube-api-access-k8w4v\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:30.261578 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:30.261533 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e8de4ed-6171-40e3-93d7-4f057f5209c2-maas-api-xks-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:30.261578 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:30.261543 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e8de4ed-6171-40e3-93d7-4f057f5209c2-maas-api-openshift-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:30.947047 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:30.947015 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vnd4z" Nov 03 20:57:30.947047 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:30.947025 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vnd4z" event={"ID":"3e8de4ed-6171-40e3-93d7-4f057f5209c2","Type":"ContainerDied","Data":"0b389fee9a97f719737032ac2ec9ef8c1c2aed3f78182c4acb996d669cc8f991"} Nov 03 20:57:30.947358 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:30.947084 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b389fee9a97f719737032ac2ec9ef8c1c2aed3f78182c4acb996d669cc8f991" Nov 03 20:57:42.815050 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:42.814951 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2"] Nov 03 20:57:42.815554 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:42.815407 4918 cpu_manager.go:414] "containerMap: removing container" podUID="3e8de4ed-6171-40e3-93d7-4f057f5209c2" containerName="revoke-keys" Nov 03 20:57:42.815554 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:42.815421 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e8de4ed-6171-40e3-93d7-4f057f5209c2" containerName="revoke-keys" Nov 03 20:57:42.815554 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:42.815507 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e8de4ed-6171-40e3-93d7-4f057f5209c2" containerName="revoke-keys" Nov 03 20:57:42.818629 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:42.818606 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2" Nov 03 20:57:42.832815 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:42.832789 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:57:42.833059 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:42.833037 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:57:42.833059 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:42.833048 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xhgp6\"" Nov 03 20:57:42.833280 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:42.833092 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:57:42.835727 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:42.835702 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2"] Nov 03 20:57:42.913187 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:42.913135 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xkwj\" (UniqueName: \"kubernetes.io/projected/5c3586cd-8120-4991-8cd2-9b0b106ef110-kube-api-access-5xkwj\") pod \"maas-api-revoke-sub-0165d7d2-zxdn2\" (UID: \"5c3586cd-8120-4991-8cd2-9b0b106ef110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2" Nov 03 20:57:42.913368 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:42.913208 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3586cd-8120-4991-8cd2-9b0b106ef110-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zxdn2\" (UID: \"5c3586cd-8120-4991-8cd2-9b0b106ef110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2" Nov 03 20:57:42.913368 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:42.913258 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3586cd-8120-4991-8cd2-9b0b106ef110-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zxdn2\" (UID: \"5c3586cd-8120-4991-8cd2-9b0b106ef110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2" Nov 03 20:57:43.013942 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:43.013904 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xkwj\" (UniqueName: \"kubernetes.io/projected/5c3586cd-8120-4991-8cd2-9b0b106ef110-kube-api-access-5xkwj\") pod \"maas-api-revoke-sub-0165d7d2-zxdn2\" (UID: \"5c3586cd-8120-4991-8cd2-9b0b106ef110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2" Nov 03 20:57:43.013942 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:43.013946 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3586cd-8120-4991-8cd2-9b0b106ef110-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zxdn2\" (UID: \"5c3586cd-8120-4991-8cd2-9b0b106ef110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2" Nov 03 20:57:43.014301 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:43.013981 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3586cd-8120-4991-8cd2-9b0b106ef110-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zxdn2\" (UID: \"5c3586cd-8120-4991-8cd2-9b0b106ef110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2" Nov 03 20:57:43.014727 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:43.014702 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3586cd-8120-4991-8cd2-9b0b106ef110-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zxdn2\" (UID: \"5c3586cd-8120-4991-8cd2-9b0b106ef110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2" Nov 03 20:57:43.016889 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:43.016861 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3586cd-8120-4991-8cd2-9b0b106ef110-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zxdn2\" (UID: \"5c3586cd-8120-4991-8cd2-9b0b106ef110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2" Nov 03 20:57:43.026149 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:43.026116 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xkwj\" (UniqueName: \"kubernetes.io/projected/5c3586cd-8120-4991-8cd2-9b0b106ef110-kube-api-access-5xkwj\") pod \"maas-api-revoke-sub-0165d7d2-zxdn2\" (UID: \"5c3586cd-8120-4991-8cd2-9b0b106ef110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2" Nov 03 20:57:43.143004 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:43.142915 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2" Nov 03 20:57:43.418098 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:43.418072 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2"] Nov 03 20:57:43.420500 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:57:43.420466 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3586cd_8120_4991_8cd2_9b0b106ef110.slice/crio-5cf0fbe5a929bbe8d7d2e88f083880ef3a8fb377299e6425221a3bf602a541a3 WatchSource:0}: Error finding container 5cf0fbe5a929bbe8d7d2e88f083880ef3a8fb377299e6425221a3bf602a541a3: Status 404 returned error can't find the container with id 5cf0fbe5a929bbe8d7d2e88f083880ef3a8fb377299e6425221a3bf602a541a3 Nov 03 20:57:44.026652 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:44.026574 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2" event={"ID":"5c3586cd-8120-4991-8cd2-9b0b106ef110","Type":"ContainerStarted","Data":"6828b1ef718817020d0fac0362ccd8c284d19ab6da340fd2c646c407a5a1cc51"} Nov 03 20:57:44.026652 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:44.026627 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2" event={"ID":"5c3586cd-8120-4991-8cd2-9b0b106ef110","Type":"ContainerStarted","Data":"5cf0fbe5a929bbe8d7d2e88f083880ef3a8fb377299e6425221a3bf602a541a3"} Nov 03 20:57:44.048648 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:44.048601 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2" podStartSLOduration=2.04858775 podStartE2EDuration="2.04858775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:44.046717352 +0000 UTC m=+3646.978584864" watchObservedRunningTime="2026-11-03 20:57:44.04858775 +0000 UTC m=+3646.980455255" Nov 03 20:57:45.033024 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:45.032989 4918 generic.go:356] "Generic (PLEG): container finished" podID="5c3586cd-8120-4991-8cd2-9b0b106ef110" containerID="6828b1ef718817020d0fac0362ccd8c284d19ab6da340fd2c646c407a5a1cc51" exitCode=0 Nov 03 20:57:45.033549 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:45.033075 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2" event={"ID":"5c3586cd-8120-4991-8cd2-9b0b106ef110","Type":"ContainerDied","Data":"6828b1ef718817020d0fac0362ccd8c284d19ab6da340fd2c646c407a5a1cc51"} Nov 03 20:57:46.173123 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:46.173097 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2" Nov 03 20:57:46.240723 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:46.240690 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c3586cd-8120-4991-8cd2-9b0b106ef110-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3586cd-8120-4991-8cd2-9b0b106ef110-maas-api-xks-ca\") pod \"5c3586cd-8120-4991-8cd2-9b0b106ef110\" (UID: \"5c3586cd-8120-4991-8cd2-9b0b106ef110\") " Nov 03 20:57:46.240895 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:46.240793 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c3586cd-8120-4991-8cd2-9b0b106ef110-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3586cd-8120-4991-8cd2-9b0b106ef110-maas-api-openshift-ca\") pod \"5c3586cd-8120-4991-8cd2-9b0b106ef110\" (UID: \"5c3586cd-8120-4991-8cd2-9b0b106ef110\") " Nov 03 20:57:46.240895 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:46.240828 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c3586cd-8120-4991-8cd2-9b0b106ef110-kube-api-access-5xkwj\" (UniqueName: \"kubernetes.io/projected/5c3586cd-8120-4991-8cd2-9b0b106ef110-kube-api-access-5xkwj\") pod \"5c3586cd-8120-4991-8cd2-9b0b106ef110\" (UID: \"5c3586cd-8120-4991-8cd2-9b0b106ef110\") " Nov 03 20:57:46.241244 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:46.241214 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c3586cd-8120-4991-8cd2-9b0b106ef110-maas-api-openshift-ca" pod "5c3586cd-8120-4991-8cd2-9b0b106ef110" (UID: "5c3586cd-8120-4991-8cd2-9b0b106ef110"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:46.242769 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:46.242732 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c3586cd-8120-4991-8cd2-9b0b106ef110-maas-api-xks-ca" pod "5c3586cd-8120-4991-8cd2-9b0b106ef110" (UID: "5c3586cd-8120-4991-8cd2-9b0b106ef110"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:57:46.242969 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:46.242950 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3586cd-8120-4991-8cd2-9b0b106ef110-kube-api-access-5xkwj" pod "5c3586cd-8120-4991-8cd2-9b0b106ef110" (UID: "5c3586cd-8120-4991-8cd2-9b0b106ef110"). InnerVolumeSpecName "kube-api-access-5xkwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:46.342433 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:46.342400 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3586cd-8120-4991-8cd2-9b0b106ef110-maas-api-xks-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:46.342433 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:46.342431 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3586cd-8120-4991-8cd2-9b0b106ef110-maas-api-openshift-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:46.342627 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:46.342441 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xkwj\" (UniqueName: \"kubernetes.io/projected/5c3586cd-8120-4991-8cd2-9b0b106ef110-kube-api-access-5xkwj\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:47.043790 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:47.043746 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2" event={"ID":"5c3586cd-8120-4991-8cd2-9b0b106ef110","Type":"ContainerDied","Data":"5cf0fbe5a929bbe8d7d2e88f083880ef3a8fb377299e6425221a3bf602a541a3"} Nov 03 20:57:47.043992 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:47.043804 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cf0fbe5a929bbe8d7d2e88f083880ef3a8fb377299e6425221a3bf602a541a3" Nov 03 20:57:47.043992 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:47.043760 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zxdn2" Nov 03 20:57:53.656948 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:53.656908 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pk2hn"] Nov 03 20:57:53.657383 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:53.657368 4918 cpu_manager.go:414] "containerMap: removing container" podUID="5c3586cd-8120-4991-8cd2-9b0b106ef110" containerName="revoke-keys" Nov 03 20:57:53.657440 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:53.657385 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c3586cd-8120-4991-8cd2-9b0b106ef110" containerName="revoke-keys" Nov 03 20:57:53.657479 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:53.657452 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c3586cd-8120-4991-8cd2-9b0b106ef110" containerName="revoke-keys" Nov 03 20:57:53.660831 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:53.660813 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pk2hn" Nov 03 20:57:53.669867 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:53.669840 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xhgp6\"" Nov 03 20:57:53.670119 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:53.670101 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:57:53.670303 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:53.670288 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:57:53.670413 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:53.670292 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:57:53.671979 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:53.671897 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pk2hn"] Nov 03 20:57:53.710508 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:53.710477 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-pk2hn\" (UID: \"ca65bceb-8bf7-4b4d-ab05-fdc9c041029b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pk2hn" Nov 03 20:57:53.710663 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:53.710548 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-pk2hn\" (UID: \"ca65bceb-8bf7-4b4d-ab05-fdc9c041029b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pk2hn" Nov 03 20:57:53.710711 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:53.710679 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5578q\" (UniqueName: \"kubernetes.io/projected/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-kube-api-access-5578q\") pod \"maas-api-revoke-sub-6b19338f-pk2hn\" (UID: \"ca65bceb-8bf7-4b4d-ab05-fdc9c041029b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pk2hn" Nov 03 20:57:53.811777 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:53.811743 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-pk2hn\" (UID: \"ca65bceb-8bf7-4b4d-ab05-fdc9c041029b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pk2hn" Nov 03 20:57:53.811777 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:53.811780 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5578q\" (UniqueName: \"kubernetes.io/projected/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-kube-api-access-5578q\") pod \"maas-api-revoke-sub-6b19338f-pk2hn\" (UID: \"ca65bceb-8bf7-4b4d-ab05-fdc9c041029b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pk2hn" Nov 03 20:57:53.812044 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:53.811841 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-pk2hn\" (UID: \"ca65bceb-8bf7-4b4d-ab05-fdc9c041029b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pk2hn" Nov 03 20:57:53.812411 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:53.812383 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-pk2hn\" (UID: \"ca65bceb-8bf7-4b4d-ab05-fdc9c041029b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pk2hn" Nov 03 20:57:53.814166 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:53.814141 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-pk2hn\" (UID: \"ca65bceb-8bf7-4b4d-ab05-fdc9c041029b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pk2hn" Nov 03 20:57:53.824154 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:53.824130 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5578q\" (UniqueName: \"kubernetes.io/projected/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-kube-api-access-5578q\") pod \"maas-api-revoke-sub-6b19338f-pk2hn\" (UID: \"ca65bceb-8bf7-4b4d-ab05-fdc9c041029b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pk2hn" Nov 03 20:57:53.979233 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:53.979103 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pk2hn" Nov 03 20:57:54.266244 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:54.266138 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pk2hn"] Nov 03 20:57:54.269374 ip-10-0-139-53 kubenswrapper[4918]: W1103 20:57:54.269339 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca65bceb_8bf7_4b4d_ab05_fdc9c041029b.slice/crio-67aaf47dcbfe629fa6c70bdd39b0f23d98f9bda6dce1cc4cebdaed9b0e3b829b WatchSource:0}: Error finding container 67aaf47dcbfe629fa6c70bdd39b0f23d98f9bda6dce1cc4cebdaed9b0e3b829b: Status 404 returned error can't find the container with id 67aaf47dcbfe629fa6c70bdd39b0f23d98f9bda6dce1cc4cebdaed9b0e3b829b Nov 03 20:57:55.084617 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:55.084577 4918 generic.go:356] "Generic (PLEG): container finished" podID="ca65bceb-8bf7-4b4d-ab05-fdc9c041029b" containerID="543b5056aef936f4171e7c7678ff857f2ca8d247e0ddd56038898582d54c1209" exitCode=0 Nov 03 20:57:55.084617 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:55.084614 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pk2hn" event={"ID":"ca65bceb-8bf7-4b4d-ab05-fdc9c041029b","Type":"ContainerDied","Data":"543b5056aef936f4171e7c7678ff857f2ca8d247e0ddd56038898582d54c1209"} Nov 03 20:57:55.085125 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:55.084687 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pk2hn" event={"ID":"ca65bceb-8bf7-4b4d-ab05-fdc9c041029b","Type":"ContainerStarted","Data":"67aaf47dcbfe629fa6c70bdd39b0f23d98f9bda6dce1cc4cebdaed9b0e3b829b"} Nov 03 20:57:56.228957 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:56.228930 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pk2hn" Nov 03 20:57:56.333044 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:56.333013 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-maas-api-xks-ca\") pod \"ca65bceb-8bf7-4b4d-ab05-fdc9c041029b\" (UID: \"ca65bceb-8bf7-4b4d-ab05-fdc9c041029b\") " Nov 03 20:57:56.333302 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:56.333087 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-kube-api-access-5578q\" (UniqueName: \"kubernetes.io/projected/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-kube-api-access-5578q\") pod \"ca65bceb-8bf7-4b4d-ab05-fdc9c041029b\" (UID: \"ca65bceb-8bf7-4b4d-ab05-fdc9c041029b\") " Nov 03 20:57:56.333302 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:56.333158 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-maas-api-openshift-ca\") pod \"ca65bceb-8bf7-4b4d-ab05-fdc9c041029b\" (UID: \"ca65bceb-8bf7-4b4d-ab05-fdc9c041029b\") " Nov 03 20:57:56.333592 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:56.333554 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-maas-api-openshift-ca" pod "ca65bceb-8bf7-4b4d-ab05-fdc9c041029b" (UID: "ca65bceb-8bf7-4b4d-ab05-fdc9c041029b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:56.335288 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:56.335263 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-maas-api-xks-ca" pod "ca65bceb-8bf7-4b4d-ab05-fdc9c041029b" (UID: "ca65bceb-8bf7-4b4d-ab05-fdc9c041029b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:57:56.335428 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:56.335411 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-kube-api-access-5578q" pod "ca65bceb-8bf7-4b4d-ab05-fdc9c041029b" (UID: "ca65bceb-8bf7-4b4d-ab05-fdc9c041029b"). InnerVolumeSpecName "kube-api-access-5578q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:56.434525 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:56.434488 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-maas-api-xks-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:56.434525 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:56.434524 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5578q\" (UniqueName: \"kubernetes.io/projected/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-kube-api-access-5578q\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:56.434716 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:56.434542 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca65bceb-8bf7-4b4d-ab05-fdc9c041029b-maas-api-openshift-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:57.096294 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:57.096258 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pk2hn" Nov 03 20:57:57.096294 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:57.096269 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pk2hn" event={"ID":"ca65bceb-8bf7-4b4d-ab05-fdc9c041029b","Type":"ContainerDied","Data":"67aaf47dcbfe629fa6c70bdd39b0f23d98f9bda6dce1cc4cebdaed9b0e3b829b"} Nov 03 20:57:57.096516 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:57:57.096325 4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67aaf47dcbfe629fa6c70bdd39b0f23d98f9bda6dce1cc4cebdaed9b0e3b829b" Nov 03 20:57:58.207490 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:57:58.207444 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:58:04.975859 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:04.975813 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c44549c46-lcdq6_8b5289a1-1b79-4406-9b5b-a354caae1c97/authorino/0.log" Nov 03 20:58:16.032722 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:16.032679 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dbzn5_60e07b0f-b78b-4f51-9ead-028f98aec0c9/manager/0.log" Nov 03 20:58:18.304578 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:18.304551 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-zxdn2_5c3586cd-8120-4991-8cd2-9b0b106ef110/revoke-keys/0.log" Nov 03 20:58:18.838295 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:18.838258 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-pk2hn_ca65bceb-8bf7-4b4d-ab05-fdc9c041029b/revoke-keys/0.log" Nov 03 20:58:19.184875 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:19.184801 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-vnd4z_3e8de4ed-6171-40e3-93d7-4f057f5209c2/revoke-keys/0.log" Nov 03 20:58:20.872109 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:20.872070 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c44549c46-lcdq6_8b5289a1-1b79-4406-9b5b-a354caae1c97/authorino/0.log" Nov 03 20:58:23.246247 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:23.246208 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-667774d4b7-l8npv_0000aa42-8c87-45d2-990d-c69f4d9c0ff9/kube-auth-proxy/0.log" Nov 03 20:58:23.616836 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:23.616804 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-8bvn4_ea819fe0-461b-4952-a40e-7a47e8f51580/istio-proxy/0.log" Nov 03 20:58:24.576258 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:24.576220 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-nwfzq_7f1582e1-cf7e-448f-b9aa-73f1badd5c6b/istio-proxy/0.log" Nov 03 20:58:24.938724 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:24.938624 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-t7zgq_ea43d6b7-986b-4194-b9b3-b86e22e86544/istio-proxy/0.log" Nov 03 20:58:25.857279 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:25.857228 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-h8g6d_cb824440-b2b0-478f-a06d-acd897fceb6d/payload-processing/0.log" Nov 03 20:58:26.043488 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:26.043452 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6bfcd89b85-9qb4s_653cb377-919b-4793-85a4-0d60d6aeaff6/router/0.log" Nov 03 20:58:27.025575 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:27.025522 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4_d51a14a7-ad15-477a-8eb6-83b5fc798851/main/0.log" Nov 03 20:58:27.198249 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:27.198219 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-vgcx6_e1a63d7a-3f9e-46dc-8e39-144eb440c4b5/main/0.log" Nov 03 20:58:27.385195 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:27.385141 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-7qnhr_34a102a4-f391-4d65-8a93-324a300363cc/main/0.log" Nov 03 20:58:27.566128 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:27.566091 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-hn84m_190167d8-3e24-427f-86d2-a53d510603ed/main/0.log" Nov 03 20:58:27.931554 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:27.931519 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-bzdwr_fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a/main/0.log" Nov 03 20:58:28.122922 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:28.122888 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk_c9493241-95b1-4b6d-9290-c677b2a55000/main/0.log" Nov 03 20:58:28.208019 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:58:28.207938 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:58:28.304033 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:28.303989 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9_8ad604d4-e127-4bd4-9d38-6f8b3987b313/main/0.log" Nov 03 20:58:29.172134 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:29.172098 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c44549c46-lcdq6_8b5289a1-1b79-4406-9b5b-a354caae1c97/authorino/0.log" Nov 03 20:58:38.882960 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:38.882927 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dbzn5_60e07b0f-b78b-4f51-9ead-028f98aec0c9/manager/0.log" Nov 03 20:58:40.837631 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:40.837602 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-zxdn2_5c3586cd-8120-4991-8cd2-9b0b106ef110/revoke-keys/0.log" Nov 03 20:58:41.212108 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:41.212021 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-pk2hn_ca65bceb-8bf7-4b4d-ab05-fdc9c041029b/revoke-keys/0.log" Nov 03 20:58:41.599517 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:41.599489 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-vnd4z_3e8de4ed-6171-40e3-93d7-4f057f5209c2/revoke-keys/0.log" Nov 03 20:58:42.941238 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:42.941198 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c44549c46-lcdq6_8b5289a1-1b79-4406-9b5b-a354caae1c97/authorino/0.log" Nov 03 20:58:45.120240 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:45.120204 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-667774d4b7-l8npv_0000aa42-8c87-45d2-990d-c69f4d9c0ff9/kube-auth-proxy/0.log" Nov 03 20:58:45.495979 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:45.495895 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-8bvn4_ea819fe0-461b-4952-a40e-7a47e8f51580/istio-proxy/0.log" Nov 03 20:58:46.486309 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:46.486271 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-nwfzq_7f1582e1-cf7e-448f-b9aa-73f1badd5c6b/istio-proxy/0.log" Nov 03 20:58:46.856238 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:46.856206 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-t7zgq_ea43d6b7-986b-4194-b9b3-b86e22e86544/istio-proxy/0.log" Nov 03 20:58:47.783058 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:47.783020 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-h8g6d_cb824440-b2b0-478f-a06d-acd897fceb6d/payload-processing/0.log" Nov 03 20:58:47.952592 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:47.952558 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6bfcd89b85-9qb4s_653cb377-919b-4793-85a4-0d60d6aeaff6/router/0.log" Nov 03 20:58:48.496339 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:48.496301 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-2tnc4_d51a14a7-ad15-477a-8eb6-83b5fc798851/main/0.log" Nov 03 20:58:48.680888 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:48.680860 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-vgcx6_e1a63d7a-3f9e-46dc-8e39-144eb440c4b5/main/0.log" Nov 03 20:58:48.868502 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:48.868477 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-7qnhr_34a102a4-f391-4d65-8a93-324a300363cc/main/0.log" Nov 03 20:58:49.051955 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:49.051922 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-hn84m_190167d8-3e24-427f-86d2-a53d510603ed/main/0.log" Nov 03 20:58:49.423375 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:49.423343 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-bzdwr_fa956d00-6eba-4ef7-bbb8-d63ed2dbdb9a/main/0.log" Nov 03 20:58:49.608168 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:49.608135 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-8lgxk_c9493241-95b1-4b6d-9290-c677b2a55000/main/0.log" Nov 03 20:58:49.781384 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:58:49.781281 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-ll7w9_8ad604d4-e127-4bd4-9d38-6f8b3987b313/main/0.log" Nov 03 20:58:58.207500 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:58:58.207462 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:59:28.208222 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:59:28.208166 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 20:59:30.362839 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:59:30.362811 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-h8g6d_cb824440-b2b0-478f-a06d-acd897fceb6d/payload-processing/0.log" Nov 03 20:59:30.551313 ip-10-0-139-53 kubenswrapper[4918]: I1103 20:59:30.551282 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-h8g6d_cb824440-b2b0-478f-a06d-acd897fceb6d/payload-processing/0.log" Nov 03 20:59:58.207935 ip-10-0-139-53 kubenswrapper[4918]: E1103 20:59:58.207898 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 21:00:00.154763 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:00.154726 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np"] Nov 03 21:00:00.155286 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:00.155172 4918 cpu_manager.go:414] "containerMap: removing container" podUID="ca65bceb-8bf7-4b4d-ab05-fdc9c041029b" containerName="revoke-keys" Nov 03 21:00:00.155286 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:00.155199 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca65bceb-8bf7-4b4d-ab05-fdc9c041029b" containerName="revoke-keys" Nov 03 21:00:00.155370 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:00.155297 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca65bceb-8bf7-4b4d-ab05-fdc9c041029b" containerName="revoke-keys" Nov 03 21:00:00.158419 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:00.158400 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np" Nov 03 21:00:00.164064 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:00.164037 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xhgp6\"" Nov 03 21:00:00.164236 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:00.164068 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:00:00.164236 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:00.164123 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:00:00.177645 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:00.177614 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np"] Nov 03 21:00:00.230509 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:00.230471 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81e3c493-cd83-47f7-9539-0f57410bf236-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-nq4np\" (UID: \"81e3c493-cd83-47f7-9539-0f57410bf236\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np" Nov 03 21:00:00.230692 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:00.230642 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97jth\" (UniqueName: \"kubernetes.io/projected/81e3c493-cd83-47f7-9539-0f57410bf236-kube-api-access-97jth\") pod \"maas-api-key-cleanup-29895660-nq4np\" (UID: \"81e3c493-cd83-47f7-9539-0f57410bf236\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np" Nov 03 21:00:00.331602 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:00.331556 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81e3c493-cd83-47f7-9539-0f57410bf236-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-nq4np\" (UID: \"81e3c493-cd83-47f7-9539-0f57410bf236\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np" Nov 03 21:00:00.331826 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:00.331690 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97jth\" (UniqueName: \"kubernetes.io/projected/81e3c493-cd83-47f7-9539-0f57410bf236-kube-api-access-97jth\") pod \"maas-api-key-cleanup-29895660-nq4np\" (UID: \"81e3c493-cd83-47f7-9539-0f57410bf236\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np" Nov 03 21:00:00.332244 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:00.332213 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81e3c493-cd83-47f7-9539-0f57410bf236-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-nq4np\" (UID: \"81e3c493-cd83-47f7-9539-0f57410bf236\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np" Nov 03 21:00:00.344894 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:00.344860 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97jth\" (UniqueName: \"kubernetes.io/projected/81e3c493-cd83-47f7-9539-0f57410bf236-kube-api-access-97jth\") pod \"maas-api-key-cleanup-29895660-nq4np\" (UID: \"81e3c493-cd83-47f7-9539-0f57410bf236\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np" Nov 03 21:00:00.471747 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:00.471668 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np" Nov 03 21:00:00.745711 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:00.745681 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np"] Nov 03 21:00:00.747704 ip-10-0-139-53 kubenswrapper[4918]: W1103 21:00:00.747671 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e3c493_cd83_47f7_9539_0f57410bf236.slice/crio-4d52eca6eb950c22df544555495e5774b85e6c89a2f110c27af9902047d1745c WatchSource:0}: Error finding container 4d52eca6eb950c22df544555495e5774b85e6c89a2f110c27af9902047d1745c: Status 404 returned error can't find the container with id 4d52eca6eb950c22df544555495e5774b85e6c89a2f110c27af9902047d1745c Nov 03 21:00:01.707972 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:01.707933 4918 generic.go:356] "Generic (PLEG): container finished" podID="81e3c493-cd83-47f7-9539-0f57410bf236" containerID="9db7b510386f52428a24168e0ca0a6a88a58086227575e15899ae4b4d44d74ab" exitCode=60 Nov 03 21:00:01.708380 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:01.708021 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np" event={"ID":"81e3c493-cd83-47f7-9539-0f57410bf236","Type":"ContainerDied","Data":"9db7b510386f52428a24168e0ca0a6a88a58086227575e15899ae4b4d44d74ab"} Nov 03 21:00:01.708380 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:01.708076 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np" event={"ID":"81e3c493-cd83-47f7-9539-0f57410bf236","Type":"ContainerStarted","Data":"4d52eca6eb950c22df544555495e5774b85e6c89a2f110c27af9902047d1745c"} Nov 03 21:00:01.708380 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:01.708231 4918 scope.go:122] "RemoveContainer" containerID="9db7b510386f52428a24168e0ca0a6a88a58086227575e15899ae4b4d44d74ab" Nov 03 21:00:02.714960 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:02.714926 4918 generic.go:356] "Generic (PLEG): container finished" podID="81e3c493-cd83-47f7-9539-0f57410bf236" containerID="2859bec0d3c01c4ba870b055f24ba8f6607ab0cc0b9bb60a641ff16bcfbe688e" exitCode=60 Nov 03 21:00:02.715471 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:02.715015 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np" event={"ID":"81e3c493-cd83-47f7-9539-0f57410bf236","Type":"ContainerDied","Data":"2859bec0d3c01c4ba870b055f24ba8f6607ab0cc0b9bb60a641ff16bcfbe688e"} Nov 03 21:00:02.715471 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:02.715090 4918 scope.go:122] "RemoveContainer" containerID="9db7b510386f52428a24168e0ca0a6a88a58086227575e15899ae4b4d44d74ab" Nov 03 21:00:02.715471 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:02.715283 4918 scope.go:122] "RemoveContainer" containerID="2859bec0d3c01c4ba870b055f24ba8f6607ab0cc0b9bb60a641ff16bcfbe688e" Nov 03 21:00:02.715587 ip-10-0-139-53 kubenswrapper[4918]: E1103 21:00:02.715563 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-nq4np_odh-ai-gateway-infra(81e3c493-cd83-47f7-9539-0f57410bf236)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np" podUID="81e3c493-cd83-47f7-9539-0f57410bf236" Nov 03 21:00:03.721987 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:03.721956 4918 scope.go:122] "RemoveContainer" containerID="2859bec0d3c01c4ba870b055f24ba8f6607ab0cc0b9bb60a641ff16bcfbe688e" Nov 03 21:00:03.722430 ip-10-0-139-53 kubenswrapper[4918]: E1103 21:00:03.722226 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-nq4np_odh-ai-gateway-infra(81e3c493-cd83-47f7-9539-0f57410bf236)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np" podUID="81e3c493-cd83-47f7-9539-0f57410bf236" Nov 03 21:00:14.207203 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:14.207148 4918 scope.go:122] "RemoveContainer" containerID="2859bec0d3c01c4ba870b055f24ba8f6607ab0cc0b9bb60a641ff16bcfbe688e" Nov 03 21:00:14.779837 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:14.779750 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np" event={"ID":"81e3c493-cd83-47f7-9539-0f57410bf236","Type":"ContainerStarted","Data":"b54a037fb50826cad046eefc94daa064729cef35286d5441f970b2f4f06e8ab5"} Nov 03 21:00:14.798054 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:14.798008 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np" podStartSLOduration=14.797994749 podStartE2EDuration="14.797994749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:14.796740701 +0000 UTC m=+3797.728608235" watchObservedRunningTime="2026-11-03 21:00:14.797994749 +0000 UTC m=+3797.729862292" Nov 03 21:00:15.787108 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:15.787067 4918 generic.go:356] "Generic (PLEG): container finished" podID="81e3c493-cd83-47f7-9539-0f57410bf236" containerID="b54a037fb50826cad046eefc94daa064729cef35286d5441f970b2f4f06e8ab5" exitCode=60 Nov 03 21:00:15.787616 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:15.787140 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np" event={"ID":"81e3c493-cd83-47f7-9539-0f57410bf236","Type":"ContainerDied","Data":"b54a037fb50826cad046eefc94daa064729cef35286d5441f970b2f4f06e8ab5"} Nov 03 21:00:15.787616 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:15.787237 4918 scope.go:122] "RemoveContainer" containerID="2859bec0d3c01c4ba870b055f24ba8f6607ab0cc0b9bb60a641ff16bcfbe688e" Nov 03 21:00:15.787616 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:15.787581 4918 scope.go:122] "RemoveContainer" containerID="b54a037fb50826cad046eefc94daa064729cef35286d5441f970b2f4f06e8ab5" Nov 03 21:00:15.787885 ip-10-0-139-53 kubenswrapper[4918]: E1103 21:00:15.787861 4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-nq4np_odh-ai-gateway-infra(81e3c493-cd83-47f7-9539-0f57410bf236)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np" podUID="81e3c493-cd83-47f7-9539-0f57410bf236" Nov 03 21:00:15.813424 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:15.813391 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np"] Nov 03 21:00:16.934102 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:16.934074 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np" Nov 03 21:00:16.979334 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:16.979306 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81e3c493-cd83-47f7-9539-0f57410bf236-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81e3c493-cd83-47f7-9539-0f57410bf236-maas-api-service-ca\") pod \"81e3c493-cd83-47f7-9539-0f57410bf236\" (UID: \"81e3c493-cd83-47f7-9539-0f57410bf236\") " Nov 03 21:00:16.979585 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:16.979349 4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e3c493-cd83-47f7-9539-0f57410bf236-kube-api-access-97jth\" (UniqueName: \"kubernetes.io/projected/81e3c493-cd83-47f7-9539-0f57410bf236-kube-api-access-97jth\") pod \"81e3c493-cd83-47f7-9539-0f57410bf236\" (UID: \"81e3c493-cd83-47f7-9539-0f57410bf236\") " Nov 03 21:00:16.979706 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:16.979673 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e3c493-cd83-47f7-9539-0f57410bf236-maas-api-service-ca" pod "81e3c493-cd83-47f7-9539-0f57410bf236" (UID: "81e3c493-cd83-47f7-9539-0f57410bf236"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:16.981737 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:16.981710 4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e3c493-cd83-47f7-9539-0f57410bf236-kube-api-access-97jth" pod "81e3c493-cd83-47f7-9539-0f57410bf236" (UID: "81e3c493-cd83-47f7-9539-0f57410bf236"). InnerVolumeSpecName "kube-api-access-97jth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:17.080845 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:17.080812 4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81e3c493-cd83-47f7-9539-0f57410bf236-maas-api-service-ca\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:17.080845 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:17.080845 4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97jth\" (UniqueName: \"kubernetes.io/projected/81e3c493-cd83-47f7-9539-0f57410bf236-kube-api-access-97jth\") on node \"ip-10-0-139-53.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:17.800873 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:17.800840 4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np" Nov 03 21:00:17.801072 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:17.800873 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np" event={"ID":"81e3c493-cd83-47f7-9539-0f57410bf236","Type":"ContainerDied","Data":"4d52eca6eb950c22df544555495e5774b85e6c89a2f110c27af9902047d1745c"} Nov 03 21:00:17.801072 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:17.800931 4918 scope.go:122] "RemoveContainer" containerID="b54a037fb50826cad046eefc94daa064729cef35286d5441f970b2f4f06e8ab5" Nov 03 21:00:17.818540 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:17.818516 4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np"] Nov 03 21:00:17.824613 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:17.824591 4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-nq4np"] Nov 03 21:00:19.213254 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:19.213221 4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81e3c493-cd83-47f7-9539-0f57410bf236" path="/var/lib/kubelet/pods/81e3c493-cd83-47f7-9539-0f57410bf236/volumes" Nov 03 21:00:28.207435 ip-10-0-139-53 kubenswrapper[4918]: E1103 21:00:28.207397 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 21:00:30.541099 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:30.541069 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-h8g6d_cb824440-b2b0-478f-a06d-acd897fceb6d/payload-processing/0.log" Nov 03 21:00:30.728039 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:30.728003 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-h8g6d_cb824440-b2b0-478f-a06d-acd897fceb6d/payload-processing/0.log" Nov 03 21:00:31.047329 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.047290 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zpp9v/must-gather-lnchb"] Nov 03 21:00:31.047690 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.047676 4918 cpu_manager.go:414] "containerMap: removing container" podUID="81e3c493-cd83-47f7-9539-0f57410bf236" containerName="cleanup" Nov 03 21:00:31.047751 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.047691 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e3c493-cd83-47f7-9539-0f57410bf236" containerName="cleanup" Nov 03 21:00:31.047751 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.047703 4918 cpu_manager.go:414] "containerMap: removing container" podUID="81e3c493-cd83-47f7-9539-0f57410bf236" containerName="cleanup" Nov 03 21:00:31.047751 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.047708 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e3c493-cd83-47f7-9539-0f57410bf236" containerName="cleanup" Nov 03 21:00:31.047868 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.047791 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e3c493-cd83-47f7-9539-0f57410bf236" containerName="cleanup" Nov 03 21:00:31.047868 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.047799 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e3c493-cd83-47f7-9539-0f57410bf236" containerName="cleanup" Nov 03 21:00:31.047868 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.047805 4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e3c493-cd83-47f7-9539-0f57410bf236" containerName="cleanup" Nov 03 21:00:31.047973 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.047878 4918 cpu_manager.go:414] "containerMap: removing container" podUID="81e3c493-cd83-47f7-9539-0f57410bf236" containerName="cleanup" Nov 03 21:00:31.047973 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.047884 4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e3c493-cd83-47f7-9539-0f57410bf236" containerName="cleanup" Nov 03 21:00:31.050921 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.050904 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpp9v/must-gather-lnchb" Nov 03 21:00:31.059647 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.059605 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zpp9v\"/\"openshift-service-ca.crt\"" Nov 03 21:00:31.059779 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.059707 4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zpp9v\"/\"default-dockercfg-dsb8v\"" Nov 03 21:00:31.059844 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.059827 4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zpp9v\"/\"kube-root-ca.crt\"" Nov 03 21:00:31.060776 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.060751 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpp9v/must-gather-lnchb"] Nov 03 21:00:31.117232 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.117165 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brbwt\" (UniqueName: \"kubernetes.io/projected/60065a04-c615-4676-a0de-facc50de25f9-kube-api-access-brbwt\") pod \"must-gather-lnchb\" (UID: \"60065a04-c615-4676-a0de-facc50de25f9\") " pod="openshift-must-gather-zpp9v/must-gather-lnchb" Nov 03 21:00:31.117385 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.117318 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/60065a04-c615-4676-a0de-facc50de25f9-must-gather-output\") pod \"must-gather-lnchb\" (UID: \"60065a04-c615-4676-a0de-facc50de25f9\") " pod="openshift-must-gather-zpp9v/must-gather-lnchb" Nov 03 21:00:31.218574 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.218544 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/60065a04-c615-4676-a0de-facc50de25f9-must-gather-output\") pod \"must-gather-lnchb\" (UID: \"60065a04-c615-4676-a0de-facc50de25f9\") " pod="openshift-must-gather-zpp9v/must-gather-lnchb" Nov 03 21:00:31.218767 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.218641 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brbwt\" (UniqueName: \"kubernetes.io/projected/60065a04-c615-4676-a0de-facc50de25f9-kube-api-access-brbwt\") pod \"must-gather-lnchb\" (UID: \"60065a04-c615-4676-a0de-facc50de25f9\") " pod="openshift-must-gather-zpp9v/must-gather-lnchb" Nov 03 21:00:31.218952 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.218928 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/60065a04-c615-4676-a0de-facc50de25f9-must-gather-output\") pod \"must-gather-lnchb\" (UID: \"60065a04-c615-4676-a0de-facc50de25f9\") " pod="openshift-must-gather-zpp9v/must-gather-lnchb" Nov 03 21:00:31.230792 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.230752 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brbwt\" (UniqueName: \"kubernetes.io/projected/60065a04-c615-4676-a0de-facc50de25f9-kube-api-access-brbwt\") pod \"must-gather-lnchb\" (UID: \"60065a04-c615-4676-a0de-facc50de25f9\") " pod="openshift-must-gather-zpp9v/must-gather-lnchb" Nov 03 21:00:31.366306 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.366263 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpp9v/must-gather-lnchb" Nov 03 21:00:31.693754 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.693729 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpp9v/must-gather-lnchb"] Nov 03 21:00:31.696246 ip-10-0-139-53 kubenswrapper[4918]: W1103 21:00:31.696221 4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60065a04_c615_4676_a0de_facc50de25f9.slice/crio-3cf79afc55030ee155ab67331f461b695cd1a9edde417a5a595a19158e8cb54c WatchSource:0}: Error finding container 3cf79afc55030ee155ab67331f461b695cd1a9edde417a5a595a19158e8cb54c: Status 404 returned error can't find the container with id 3cf79afc55030ee155ab67331f461b695cd1a9edde417a5a595a19158e8cb54c Nov 03 21:00:31.698344 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.698320 4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:00:31.875153 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:31.875108 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpp9v/must-gather-lnchb" event={"ID":"60065a04-c615-4676-a0de-facc50de25f9","Type":"ContainerStarted","Data":"3cf79afc55030ee155ab67331f461b695cd1a9edde417a5a595a19158e8cb54c"} Nov 03 21:00:32.882523 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:32.882480 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpp9v/must-gather-lnchb" event={"ID":"60065a04-c615-4676-a0de-facc50de25f9","Type":"ContainerStarted","Data":"d3e2b50b73be5d2f1e421c114792bf570277cf74f65b2d2df0da1aae65a2a036"} Nov 03 21:00:32.882975 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:32.882535 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpp9v/must-gather-lnchb" event={"ID":"60065a04-c615-4676-a0de-facc50de25f9","Type":"ContainerStarted","Data":"8a15ec0bf559aaf1877df079f90d28f8ab23db8581e4a72de95f39ffe8e0cbe2"} Nov 03 21:00:32.900620 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:32.900562 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zpp9v/must-gather-lnchb" podStartSLOduration=1.123460824 podStartE2EDuration="1.90054725s" totalImagesPullingTime="777.086426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:32.900238374 +0000 UTC m=+3815.832105885" watchObservedRunningTime="2026-11-03 21:00:32.90054725 +0000 UTC m=+3815.832414760" Nov 03 21:00:34.666967 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:34.666930 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mvsdz_d2e4493e-d29f-49b6-851e-cbd01214bf5c/global-pull-secret-syncer/0.log" Nov 03 21:00:34.717877 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:34.717848 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5dgdg_61f1d7a9-a8b3-4db5-878d-4b8d1632fafc/connection-checker/0.log" Nov 03 21:00:34.743993 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:34.743964 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6r7g9_158d8c01-2d31-43b4-a098-0c731c1fd163/connection-checker/0.log" Nov 03 21:00:34.769745 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:34.769715 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-qkzrh_5a7d187d-9a17-475b-8ee4-ae5836d9a73a/connection-checker/0.log" Nov 03 21:00:34.908214 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:34.907389 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ntz8w_96087edc-f246-4ba4-a9d6-734a997da1f9/konnectivity-agent/0.log" Nov 03 21:00:34.995369 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:34.995294 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-53.us-east-2.compute.internal_cb28595e81de1584c6e009472415b9b8/haproxy/0.log" Nov 03 21:00:43.837269 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:43.837167 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c44549c46-lcdq6_8b5289a1-1b79-4406-9b5b-a354caae1c97/authorino/0.log" Nov 03 21:00:43.863571 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:43.863530 4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx"] Nov 03 21:00:43.870521 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:43.870489 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" Nov 03 21:00:43.875072 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:43.875047 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx"] Nov 03 21:00:43.918034 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:43.917998 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2fnm\" (UniqueName: \"kubernetes.io/projected/4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06-kube-api-access-b2fnm\") pod \"perf-node-gather-daemonset-6chtx\" (UID: \"4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" Nov 03 21:00:43.918407 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:43.918383 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06-lib-modules\") pod \"perf-node-gather-daemonset-6chtx\" (UID: \"4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" Nov 03 21:00:43.918521 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:43.918421 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06-sys\") pod \"perf-node-gather-daemonset-6chtx\" (UID: \"4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" Nov 03 21:00:43.918521 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:43.918476 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06-podres\") pod \"perf-node-gather-daemonset-6chtx\" (UID: \"4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" Nov 03 21:00:43.918521 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:43.918513 4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06-proc\") pod \"perf-node-gather-daemonset-6chtx\" (UID: \"4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" Nov 03 21:00:44.019325 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:44.019285 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06-lib-modules\") pod \"perf-node-gather-daemonset-6chtx\" (UID: \"4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" Nov 03 21:00:44.019549 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:44.019337 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06-sys\") pod \"perf-node-gather-daemonset-6chtx\" (UID: \"4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" Nov 03 21:00:44.019549 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:44.019385 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06-podres\") pod \"perf-node-gather-daemonset-6chtx\" (UID: \"4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" Nov 03 21:00:44.019549 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:44.019419 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06-proc\") pod \"perf-node-gather-daemonset-6chtx\" (UID: \"4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" Nov 03 21:00:44.019549 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:44.019467 4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2fnm\" (UniqueName: \"kubernetes.io/projected/4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06-kube-api-access-b2fnm\") pod \"perf-node-gather-daemonset-6chtx\" (UID: \"4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" Nov 03 21:00:44.019549 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:44.019522 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06-podres\") pod \"perf-node-gather-daemonset-6chtx\" (UID: \"4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" Nov 03 21:00:44.019549 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:44.019464 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06-lib-modules\") pod \"perf-node-gather-daemonset-6chtx\" (UID: \"4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" Nov 03 21:00:44.019897 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:44.019675 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06-sys\") pod \"perf-node-gather-daemonset-6chtx\" (UID: \"4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" Nov 03 21:00:44.019897 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:44.019724 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06-proc\") pod \"perf-node-gather-daemonset-6chtx\" (UID: \"4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" Nov 03 21:00:44.048209 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:44.046090 4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2fnm\" (UniqueName: \"kubernetes.io/projected/4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06-kube-api-access-b2fnm\") pod \"perf-node-gather-daemonset-6chtx\" (UID: \"4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" Nov 03 21:00:44.193401 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:44.193304 4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" Nov 03 21:00:44.506253 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:44.502190 4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx"] Nov 03 21:00:45.078579 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:45.078519 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" event={"ID":"4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06","Type":"ContainerStarted","Data":"24841a479b468544c8440bb5a1b39663216625bb1922b772cb527ad2b08eb2cd"} Nov 03 21:00:45.079052 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:45.078637 4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" event={"ID":"4fd8af7c-34bd-4dd2-a2b8-32d20ab0bf06","Type":"ContainerStarted","Data":"a161099b3fa073451d8bc90ce196a5640bea25f54f3f4a6e0716e151bc463422"} Nov 03 21:00:45.079052 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:45.078947 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" Nov 03 21:00:45.099420 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:45.099150 4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" podStartSLOduration=2.099135773 podStartE2EDuration="2.099135773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:45.098786003 +0000 UTC m=+3828.030653496" watchObservedRunningTime="2026-11-03 21:00:45.099135773 +0000 UTC m=+3828.031003300" Nov 03 21:00:45.347486 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:45.347386 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-nb7st_4726eb4c-48e8-423d-a62c-250a18bd8b83/cluster-monitoring-operator/0.log" Nov 03 21:00:45.688985 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:45.688896 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-spdwv_4861f4b5-e2d1-474a-9f1b-dac138b37496/node-exporter/0.log" Nov 03 21:00:45.717725 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:45.717690 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-spdwv_4861f4b5-e2d1-474a-9f1b-dac138b37496/kube-rbac-proxy/0.log" Nov 03 21:00:45.742436 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:45.742406 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-spdwv_4861f4b5-e2d1-474a-9f1b-dac138b37496/init-textfile/0.log" Nov 03 21:00:47.543913 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:47.543883 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-2zqfd_ec1c4fd8-fa4e-45de-8c10-4e5745b0ca22/networking-console-plugin/0.log" Nov 03 21:00:48.112314 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:48.112277 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/2.log" Nov 03 21:00:48.118325 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:48.118292 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jk8h6_433c5b6b-176b-44d1-afc0-6884045e468c/console-operator/3.log" Nov 03 21:00:49.388896 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:49.388852 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-mvxwm_c60cb950-5dd9-4a2b-bba3-b3c38d3591e3/volume-data-source-validator/0.log" Nov 03 21:00:50.828198 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:50.828155 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xsh7l_93a3cf28-8c0c-4735-b177-d28bc7c8ea3f/dns/0.log" Nov 03 21:00:50.857078 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:50.857046 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xsh7l_93a3cf28-8c0c-4735-b177-d28bc7c8ea3f/kube-rbac-proxy/0.log" Nov 03 21:00:50.890582 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:50.890553 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ksm7q_81ec8a45-bcb9-4aff-b9d0-a6bcf9fb2f40/dns-node-resolver/0.log" Nov 03 21:00:51.096810 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:51.096719 4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-6chtx" Nov 03 21:00:51.662729 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:51.662694 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jjkqt_26ac4e9f-3445-4994-aa1b-086b39b8c1cc/node-ca/0.log" Nov 03 21:00:53.599242 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:53.599213 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-667774d4b7-l8npv_0000aa42-8c87-45d2-990d-c69f4d9c0ff9/kube-auth-proxy/0.log" Nov 03 21:00:53.675694 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:53.675665 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-8bvn4_ea819fe0-461b-4952-a40e-7a47e8f51580/istio-proxy/0.log" Nov 03 21:00:53.880571 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:53.880445 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-nwfzq_7f1582e1-cf7e-448f-b9aa-73f1badd5c6b/istio-proxy/0.log" Nov 03 21:00:53.952452 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:53.952415 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-t7zgq_ea43d6b7-986b-4194-b9b3-b86e22e86544/istio-proxy/0.log" Nov 03 21:00:54.140195 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:54.140056 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-h8g6d_cb824440-b2b0-478f-a06d-acd897fceb6d/payload-processing/0.log" Nov 03 21:00:54.180511 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:54.180465 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6bfcd89b85-9qb4s_653cb377-919b-4793-85a4-0d60d6aeaff6/router/0.log" Nov 03 21:00:55.579199 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:55.579070 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-667774d4b7-l8npv_0000aa42-8c87-45d2-990d-c69f4d9c0ff9/kube-auth-proxy/0.log" Nov 03 21:00:55.626981 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:55.626947 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-8bvn4_ea819fe0-461b-4952-a40e-7a47e8f51580/istio-proxy/0.log" Nov 03 21:00:55.731690 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:55.731646 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-nwfzq_7f1582e1-cf7e-448f-b9aa-73f1badd5c6b/istio-proxy/0.log" Nov 03 21:00:55.767365 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:55.767321 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-t7zgq_ea43d6b7-986b-4194-b9b3-b86e22e86544/istio-proxy/0.log" Nov 03 21:00:55.862797 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:55.862706 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-h8g6d_cb824440-b2b0-478f-a06d-acd897fceb6d/payload-processing/0.log" Nov 03 21:00:55.884258 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:55.884159 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6bfcd89b85-9qb4s_653cb377-919b-4793-85a4-0d60d6aeaff6/router/0.log" Nov 03 21:00:56.517029 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:56.517003 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-95k62_aed0ee7d-d648-4138-b5ef-2ae6795321d2/serve-healthcheck-canary/0.log" Nov 03 21:00:57.092752 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:57.092697 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-zl889_449a05ee-ead2-4507-8d65-faaeb3ab3281/insights-operator/0.log" Nov 03 21:00:57.094300 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:57.094264 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-zl889_449a05ee-ead2-4507-8d65-faaeb3ab3281/insights-operator/1.log" Nov 03 21:00:57.212274 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:57.212247 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k9x2f_1fc8cec5-9248-4a47-822e-d81b37a380ef/kube-rbac-proxy/0.log" Nov 03 21:00:57.236100 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:57.236069 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k9x2f_1fc8cec5-9248-4a47-822e-d81b37a380ef/exporter/0.log" Nov 03 21:00:57.263303 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:57.263273 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k9x2f_1fc8cec5-9248-4a47-822e-d81b37a380ef/extractor/0.log" Nov 03 21:00:57.384394 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:57.384321 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-rt7km-2v6j4_94b59eff-2de3-4954-8905-3d63490df350/insights-gathering/0.log" Nov 03 21:00:58.210422 ip-10-0-139-53 kubenswrapper[4918]: E1103 21:00:58.210384 4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k9x2f" containerName="extractor" Nov 03 21:00:59.086806 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:59.086776 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-667774d4b7-l8npv_0000aa42-8c87-45d2-990d-c69f4d9c0ff9/kube-auth-proxy/0.log" Nov 03 21:00:59.123623 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:59.123592 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-8bvn4_ea819fe0-461b-4952-a40e-7a47e8f51580/istio-proxy/0.log" Nov 03 21:00:59.209552 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:59.209504 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-nwfzq_7f1582e1-cf7e-448f-b9aa-73f1badd5c6b/istio-proxy/0.log" Nov 03 21:00:59.245310 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:59.245249 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-t7zgq_ea43d6b7-986b-4194-b9b3-b86e22e86544/istio-proxy/0.log" Nov 03 21:00:59.421085 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:59.420993 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-h8g6d_cb824440-b2b0-478f-a06d-acd897fceb6d/payload-processing/0.log" Nov 03 21:00:59.444018 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:59.443986 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6bfcd89b85-9qb4s_653cb377-919b-4793-85a4-0d60d6aeaff6/router/0.log" Nov 03 21:00:59.969628 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:59.969590 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-79746d584b-c2cmc_c1317f85-fde1-417b-a0aa-fe15302cfb9f/manager/0.log" Nov 03 21:00:59.995382 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:00:59.995352 4918 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-79746d584b-c2cmc_c1317f85-fde1-417b-a0aa-fe15302cfb9f/copy-manifests/0.log" Nov 03 21:01:00.384986 ip-10-0-139-53 kubenswrapper[4918]: I1103 21:01:00.384948 4918 scope.go:122] "RemoveContainer" containerID="b72174e4db0fe9303d0cd9683c45f3800d2eb8938794060813d7bc2dff705af3"